Method and system for rebuilding data in a distributed raid system

ABSTRACT

Embodiments of the systems and methods disclosed provide a distributed RAID system comprising a set of data banks. More particularly, in certain embodiments of a distributed RAID system each data bank has a set of associated storage media and executes a similar distributed RAID application. The distributed RAID applications on each of the data banks coordinate among themselves to distribute and control data flow associated with implementing a level of RAID in conjunction with a volume stored on the associated storage media of the data banks. Migration of volumes, or portions thereof, from one configuration to another configuration may be accomplished according to a priority associated with the volume.

RELATED APPLICATIONS

This application is a continuation of, and claims a benefit of priorityunder 35 U.S.C. 120 of the filing date of U.S. patent application Ser.No. 13/224,936 by inventors Galloway, et al., entitled “Method andSystem for Rebuilding Data in a Distributed RAID System” filed on Sep.2, 2011, which in turn is a continuation of, and claims a benefit ofpriority under 35 U.S.C. 120 of the filing date of U.S. patentapplication Ser. No. 12/479,434, by inventors Galloway, et al., entitled“Method and System for Rebuilding Data in a Distributed RAID System”filed on Jun. 5, 2009, now U.S. Pat. No. 8,082,393, which in turn claimsa benefit of priority under 35 U.S.C. §119 to provisional patentapplication Nos. 61/131,270 by inventors Galloway et al., entitled“Method and System for Distributed RAID Implementation” filed on Jun. 6,2008; and 61/131,314 by inventors Galloway et al., entitled “Method andSystem for Data Migration in a Distributed Multi-Processor RAIDSubsystem” filed Jun. 6, 2008; and 61/131,291 by inventors Galloway etal., entitled “System and Method for Distributing Read/Write Requests toOptimal SCSI Targets” filed Jun. 6, 2008; and 61/131,290 by inventorsGalloway et al., entitled “Method and System for Utilizing Storage in aStorage System” filed Jun. 6, 2008; and 61/131,379 by inventors Gallowayet al., entitled “Method and System for Rebuilding Data” filed Jun. 6,2008; and 61/131,312 by inventors Galloway et al., entitled “Method andSystem for Placement of Data on Storage” filed Jun. 6, 2008, the entirecontents of each are hereby fully incorporated by reference herein forall purposes.

TECHNICAL FIELD

This invention relates generally to the use of storage devices. Moreparticularly, embodiments of this invention relate to implementing RAIDon storage devices. Even more specifically, certain embodiments of thisinvention relate to a distributed implementation of RAID.

BACKGROUND

Data represents a significant asset for many entities. Consequently,data loss, whether accidental or caused by malicious activity, can becostly in terms of wasted manpower, loss of goodwill from customers,loss of time and potential legal liability. To ensure proper protectionof data for business, legal or other purposes, many entities may desireto protect their data using a variety of techniques, including datastorage, redundancy, security, etc. These techniques may, however,conflict with other competing constraints or demands imposed by thestate or configuration of computing devices used to process or storethis data.

One method for dealing with these tensions is to implement a RedundantArray of Independent Disks (RAID). Generally, RAID systems divide andreplicate data across multiple hard disk drives (or other types ofstorage media), collectively referred to as an array, to increasereliability and in some cases improve throughput of computing devices(known as a host) using these RAID systems for storage. To a host then,a RAID array may appear as one or more monolithic storage areas. When ahost desires to communicate (read, write, etc.) with the RAID system thehost communicates as if the RAID array were a single disk. The RAIDsystem, in turn, processes these communications to implement a certainRAID level in conjunction with such communications. These RAID levelsmay be designed to achieve some desired balance between a variety oftradeoffs such as reliability, capacity, speed, etc. For example, RAID(level) 0 distributes data across several disks in a way which givesimproved speed and utilizes substantially the full capacity of thedisks, but all data on a disk will be lost if the disk fails; RAID(level) 1 uses two (or more) disks which each store the same data, sothat data is not lost so long as one disk survives. Total capacity ofthe array is substantially the capacity of a single disk and RAID(level) 5 combines three or more disks in a way that protects dataagainst loss of any one disk; the storage capacity of the array isreduced by one disk.

Current implementations of RAID may have a variety of problems. Theseproblems may stem from limitations imposed by the architecture of theseRAID systems, such as the fact that in many instances all communicationswith a RAID system must be addressed to a single server which controlsand manages the RAID system. Other problems may arise from theconfiguration or layout of the data on the disks comprising a RAIDsystem. For example, in certain cases a RAID level must be chosen andstorage allocated within the RAID system before the RAID system can beutilized. Thus, the initially chosen RAID level must be implemented inconjunction with the data stored on the RAID system, irrespective ofwhether that level of RAID is desired or needed. In many cases theseexisting problems may be exacerbated by the need to use custom hardwareor software to implement these solutions, raising the costs associatedwith implementing such a solution.

Furthermore, in existing RAID systems the addition or removal ofstorage, the reconfiguration of the RAID level of various volumes or avariety of other occurrences may cause a significant disruption to theoperation of the RAID system as it may be necessary to migrate orotherwise move data to different locations, or create additional datathat was not previously extant, and these requirements may, in turn,necessitate that at least a portion of the RAID system, or data storedin that RAID system, be taken offline for a period of time. In the caseof large volumes, the time period may be hours or days. Alternatively,and equally problematic, when new storage is added to these RAID systemsthis new storage may be considered separate and distinct from existingstorage which, in turn, creates a dichotomy between original data storedin an original configuration and original storage area and data storedafter reconfiguration of storage.

Consequently, it is desired to substantially ameliorate these problems.

SUMMARY

Embodiments of the systems and methods disclosed provide a distributedRAID system comprising a set of data banks. More particularly, incertain embodiments of a distributed RAID system each data bank has aset of associated storage media and executes a similar distributed RAIDapplication. The distributed RAID applications on each of the data bankscoordinate among themselves to distribute and control data flowassociated with implementing a level of RAID in conjunction with datastored on the associated storage media of the data banks.

Specifically, in one embodiment, a volume with an associated RAID levelmay be created using the distributed RAID system. Each of thedistributed RAID applications can then coordinate operations associatedwith data of that volume such that data associated with that volume orthe implementation of the desired RAID level in conjunction with thatvolume may be stored on the multiple data banks of the distributed RAIDsystem.

Certain embodiments may be operable to migrate a volume betweenconfigurations such that data comprising a volume or correspondingredundancy data, may be created or moved among the plurality of databanks or within a single data bank, etc. By coordinating theimplementation of a level of RAID in conjunction with a volume bystoring both data of the volume and data associated with theimplementation of RAID on multiple data banks using similar distributedRAID applications executing on each of those data banks a number ofadvantages may be achieved. Namely, migration of volumes, or portionsthereof, from one configuration to another configuration may beaccomplished according to a priority associated with the volume. Bymigrating according to priorities associated with the various volumes,relatively more important volumes may be rebuilt before other relativelyless important volumes. This capability may, in turn, improveperformance with respect to these volumes more quickly in response to,for example, the failure of a disk.

These, and other, aspects of the invention will be better appreciatedand understood when considered in conjunction with the followingdescription and the accompanying drawings. The following description,while indicating various embodiments of the invention and numerousspecific details thereof, is given by way of illustration and not oflimitation. Many substitutions, modifications, additions orrearrangements may be made within the scope of the invention, and theinvention includes all such substitutions, modifications, additions orrearrangements.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings accompanying and forming part of this specification areincluded to depict certain aspects of the invention. A clearerimpression of the invention, and of the components and operation ofsystems provided with the invention, will become more readily apparentby referring to the exemplary, and therefore nonlimiting, embodimentsillustrated in the drawings, wherein identical reference numeralsdesignate the same components. Note that the features illustrated in thedrawings are not necessarily drawn to scale.

FIG. 1 is a block diagram of one embodiment of an architecture employinga distributed RAID system.

FIG. 2A is a block diagram of one embodiment of a data bank.

FIG. 2B is a block diagram of one embodiment of an architecture for adata bank.

FIG. 3 is a flow diagram of one embodiment of a method implemented by adistributed RAID system.

FIG. 4 is block diagram of an example of one embodiment of anarchitecture employing a distributed RAID system.

FIG. 5 is a block diagram of one embodiment of a table.

FIG. 6 is a block diagram of one embodiment of a table.

FIG. 7 is a block diagram of one embodiment of a table.

FIG. 8 is a flow diagram of one embodiment of a method implemented by adistributed RAID system.

FIG. 9A is a flow diagram of one embodiment of a method implemented by adistributed RAID system.

FIG. 9B is a flow diagram of one embodiment of a method implemented by adistributed RAID system.

FIG. 10 is a block diagram of one embodiment of a write cache.

FIG. 11 is a flow diagram of one embodiment of a method implemented by adistributed RAID system.

FIG. 12 is a flow diagram of one embodiment of a method implemented by adistributed RAID system.

FIG. 13 is block diagram of an example of one embodiment of anarchitecture employing a distributed RAID system.

FIG. 14 is block diagram of an example of one embodiment of an datastores in a distributed RAID system.

FIG. 15 is a flow diagram of one embodiment of a method for migrationimplemented by a distributed RAID system.

FIG. 16 is a block diagram of one embodiment of a table.

FIG. 17 is a block diagram of one embodiment of a table.

FIG. 18 is a block diagram of one embodiment of a table.

FIG. 19 is a block diagram of one embodiment of a table.

FIGS. 20A and 20B are block diagrams.

FIG. 21 is a block diagram of one embodiment of a table.

FIG. 22 is block diagram of an example of one embodiment of anarchitecture employing a distributed RAID system.

FIG. 23 is a block diagram of one embodiment of a table.

FIG. 24 is a block diagram of one embodiment of a table.

FIG. 25 is a flow diagram of one embodiment of a method for prioritizingvolumes in a migration process.

DETAILED DESCRIPTION

The invention and the various features and advantageous details thereofare explained more fully with reference to the nonlimiting embodimentsthat are illustrated in the accompanying drawings and detailed in thefollowing description. Descriptions of well known starting materials,processing techniques, components and equipment are omitted so as not tounnecessarily obscure the invention in detail. It should be understood,however, that the detailed description and the specific examples, whileindicating preferred embodiments of the invention, are given by way ofillustration only and not by way of limitation. Various substitutions,modifications, additions and/or rearrangements within the spirit and/orscope of the underlying inventive concept will become apparent to thoseskilled in the art from this disclosure. Embodiments discussed hereincan be implemented in suitable computer-executable instructions that mayreside on a computer readable medium (e.g., a HD), hardware circuitry orthe like, or any combination.

Before discussing specific embodiments, embodiments of a hardwarearchitecture for implementing certain embodiments is described herein.One embodiment can include one or more computers communicatively coupledto a network. As is known to those skilled in the art, the computer caninclude a central processing unit (“CPU”), at least one read-only memory(“ROM”), at least one random access memory (“RAM”), at least one harddrive (“HD”), and one or more input/output (“I/O”) device(s). The I/Odevices can include a keyboard, monitor, printer, electronic pointingdevice (such as a mouse, trackball, stylist, etc.), or the like. Invarious embodiments, the computer has access to at least one databaseover the network.

ROM, RAM, and HD are computer memories for storing computer-executableinstructions executable by the CPU. Within this disclosure, the term“computer-readable medium” is not limited to ROM, RAM, and HD and caninclude any type of data storage medium that can be read by a processor.In some embodiments, a computer-readable medium may refer to a datacartridge, a data backup magnetic tape, a floppy diskette, a flashmemory drive, an optical data storage drive, a CD-ROM, ROM, RAM, HD, orthe like.

At least portions of the functionalities or processes described hereincan be implemented in suitable computer-executable instructions. Thecomputer-executable instructions may be stored as software codecomponents or modules on one or more computer readable media (such asnon-volatile memories, volatile memories, DASD arrays, magnetic tapes,floppy diskettes, hard drives, optical storage devices, etc. or anyother appropriate computer-readable medium or storage device). In oneembodiment, the computer-executable instructions may include lines ofcomplied C++, Java, HTML, or any other programming or scripting code.

Additionally, the functions of the disclosed embodiments may beimplemented on one computer or shared/distributed among two or morecomputers in or across a network. Communications between computersimplementing embodiments can be accomplished using any electronic,optical, radio frequency signals, or other suitable methods and tools ofcommunication in compliance with known network protocols.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,process, article, or apparatus that comprises a list of elements is notnecessarily limited only those elements but may include other elementsnot expressly listed or inherent to such process, process, article, orapparatus. Further, unless expressly stated to the contrary, “or” refersto an inclusive or and not to an exclusive or. For example, a conditionA or B is satisfied by any one of the following: A is true (or present)and B is false (or not present), A is false (or not present) and B istrue (or present), and both A and B are true (or present).

Additionally, any examples or illustrations given herein are not to beregarded in any way as restrictions on, limits to, or expressdefinitions of, any term or terms with which they are utilized. Instead,these examples or illustrations are to be regarded as being describedwith respect to one particular embodiment and as illustrative only.Those of ordinary skill in the art will appreciate that any term orterms with which these examples or illustrations are utilized willencompass other embodiments which may or may not be given therewith orelsewhere in the specification and all such embodiments are intended tobe included within the scope of that term or terms. Language designatingsuch nonlimiting examples and illustrations includes, but is not limitedto: “for example,” “for instance,” “e.g.,” “in one embodiment”.

This application is related to U.S. patent application Ser. Nos.12/479,319, entitled “Method and System for Distributed RAIDImplementation” by Galloway et al., filed on Jun. 5, 2009; 12/479,360,entitled “Method and System for Data Migration in a Distributed RAIDImplementation” by Galloway et al., filed on Jun. 5, 2009; 12/479,403,entitled “Method and System for Distributing Commands to Targets” byGalloway et al., filed Jun. 5, 2009; 12/479,377, entitled “Method andSystem for Initializing Storage in a Storage System” by Galloway et al.,filed Jun. 5, 2009; and 12/479,394, entitled “Method and System forPlacement of Data on a Storage Device” by Galloway et al., filed Jun. 5,2009 all of which are incorporated fully herein by reference.

A brief discussion of context particularly with respect to data storagemay now be helpful. As discussed above, RAID systems divide andreplicate data across multiple hard disk drives (or other types ofstorage media), collectively referred to as an array, to increasereliability and in some cases improve throughput of computing devices(known as a host) using these RAID systems for storage. However, currentimplementations of RAID may have a variety of problems. These problemsmay stem from limitations imposed by the architecture of these RAIDsystems, from the configuration or layout of the data on the diskscomprising a RAID system, from the need to use custom hardware orsoftware to implement these solutions or other problems. It is desiredto substantially ameliorate these problems, among others.

To that end, attention is now directed to the systems and methods of thepresent invention. More specifically, embodiments of these systems andmethods provide a distributed RAID system comprising a set of databanks. Particularly, in certain embodiments of a distributed RAID systemeach data bank has a set of associated storage media and executes asimilar distributed RAID application. The distributed RAID applicationson each of the data banks coordinate among themselves to distribute andcontrol data flow associated with implementing a level of RAID inconjunction with data stored on the associated storage media of the databanks.

Specifically, in certain embodiments, a volume with an associated RAIDlevel may be created using the distributed RAID system. Each of thedistributed RAID applications can then coordinate operations associatedwith data of that volume such that data associated with that volume orthe implementation of the desired RAID level in conjunction with thatvolume may be stored on the multiple data banks of the distributed RAIDsystem.

Furthermore, alterations to such a distributed RAID system may be moreeasily dealt with. Specifically, the addition of new storage within adata bank, the failure or removal of storage within a data bank, theaddition or removal of data banks, the alteration of a RAID level orRAID type of a volume or any of a number of other causes may entail themovement of data comprising a volume, movement of redundancy datacorresponding to a volume, or the creation and placement of redundancyof data corresponding to a volume, or that data comprising a volume orredundancy data be created or moved among the plurality of data banks orwithin a single data bank (collectively referred to herein asmigration).

This migration may be seamlessly accomplished without loss of access tothe data of a volume by migrating a portion of a volume (and anyassociated redundancy data if necessary) at a time. Specifically, in oneembodiment, the portion of a volume to migrated may be segregated fromthe remainder of the volume by creating a table describing the portionof the volume to be migrated in the source (original) format and a tabledescribing the portion of the volume in the destination format (in otherwords, the format which that portion of the volume will have after themigration). These tables can then be used to handle any accesses to thatportion of the volume during the actual migration of that portion (themovement or creation of any data associated with that portion).

By using these tables during the migration of data a number ofadvantages may be achieved. Namely, storage may be seamlessly added orremoved, RAID levels or types changed, failures easily dealt with, etc.In addition, access to data being migrated may be substantiallymaintained during the process, increasing availability and reducingdowntime. Many other advantages may also be realized utilizingembodiments presented herein or other embodiments, and such advantages,which may or may not be pointed out in particular detail, will berealized after reading this disclosure.

Turning now to FIG. 1, a block diagram of an architecture for a systemwhich utilizes one embodiment of a distributed RAID system is depicted.Distributed RAID system 100 includes a set of data banks 110, each databank 110 communicatively coupled to both of switches 120. Each ofswitches 120 is also communicatively coupled to each host 102, such thata host 102 may communicate with each data bank 110 through a set ofpaths corresponding to a particular data bank 110, each path comprisingone of the switches 120.

The communicative coupling between data banks 110, switches 120 andhosts 102 may be accomplished using almost any transport medium (eitherwired or wireless) desired, including Ethernet, SCSI, iSCSI, FibreChannel, serial attached SCSI (“SAS”), advanced technology attachment(“ATA”), serial ATA (“SATA”) or other protocols known in the art.Furthermore, the communicative coupling may be implemented inconjunction with a communications network such as the Internet, a LAN, aWAN, a wireless network or any other communications network known in theart.

In one embodiment, then, using a commands protocol, such as iSCSI, SCSI,etc., hosts 102 may communicate with data banks 110 to manipulate data.More particularly, each of data banks 110 comprises storage media (aswill be explained in more detail later on herein). Collectively, thestorage media in data banks 110 may be virtualized and presented tohosts 102 as one or more contiguous blocks of storage, storage devices,etc. For example, when the iSCSI protocol is utilized the storage mediain data banks 110 may be presented to hosts 102 as a SCSI target with,in one embodiment, multiple ports.

Thus, during operation, in one embodiment a host 102 (or a user at ahost 102 or interfacing with data bank 110) may request the creation ofa volume and specify a level of RAID to be implemented in conjunctionwith that volume. Data associated with that volume and theimplementation of the desired level RAID in association with that volumeis stored across data banks 110. The hosts 102 may then access thisvolume using logical address corresponding to the volume or a portionthereof. In this manner, hosts 102 can utilize created volumes ofstorage and fault tolerance can be achieved in conjunction with thesevolumes substantially invisibly to hosts 102.

The virtualization of storage and the implementation of RAID utilizingdata banks 110 may be better understood with reference to FIG. 2A whichdepicts a block diagram of one embodiment of a data bank 110 computeroperable to implement distributed RAID. Here, data bank 110 comprises adata store 250, and a processor 202 operable to execute instructionsstored on a computer readable medium, where the instructions areoperable to implement distributed RAID application 210. Distributed RAIDapplication 210 may periodically issue heartbeat communications todistributed RAID applications 210 on other data banks 110 to determineif there has been a fault with respect to that data bank 110. If thedistributed RAID application 210 determines that another data bank 110is experiencing a fault it may set one or more fault flags correspondingto that data bank 110. Using these fault flags for each distributed RAIDapplication 210 on each data bank 110 a particular distributed RAIDapplication 210 may determine if a certain data bank 110 is faulty.

Distributed RAID application 210 may also have access (for example, toread, write, issue commands, etc.) to data store 250 comprising one ormore storage media, which may for example be disks 252 operatingaccording to almost any protocol known, such as SATA, PATA, FC, etc.where each of the disks 252 may, or may not, be of equal size.Distributed RAID application 210, executing on each of data banks 110can allow the allocation of and use of volumes using data stores 250across data banks 110 and the implementation of RAID in conjunction withthese volumes utilizing a set of global tables 240 shared between databanks 110, a set of local tables 245 and write cache 260, all of whichmay be stored in a memory 230 (which may be data store 250 or anothermemory altogether).

FIG. 2B depicts a block diagram of one embodiment of a hardwarearchitecture which may be used to implement data bank 110 computeroperable to implement distributed RAID. In this architectural example,data bank 110 comprises one or more processors 202 which may adhere tothe Intel x86 architecture or some other architecture altogether and amemory 230 coupled through a bus to I/O controller hub 212, which in oneembodiment may be a southbridge chip or the like. The I/O controller hub212 may, in turn, be coupled to and control a bus 272 such as a PCI-Xbus, PCI-express bus, etc. Coupled to this bus 272 are one or more diskcontrollers 262 such as, for example, an LSI 1068 SATA/SAS controller.Each of these disk controllers 262 is coupled to one or more disks 252,where collectively these disks 252 may comprise data store 250.Additionally, one or more network interfaces 282 may also be coupled tobus 272. These network interfaces 282 may be network interfaces (such asEthernet, etc.) which are included on motherboards, may comprise one ormore network interface cards configured to interface via one or moreprotocols such as Ethernet, fibre channel, etc. or may be some othertype of network interface such that data bank 110 may communicate withswitched 120 through these network interfaces 282.

Moving now to FIG. 3, one embodiment of a method for the allocation ofvolumes and the laying out of data associated with these volumes in datastores 250 across data banks 110 is illustrated. Initially, beforevolumes are to be allocated on a disk 252 of data store 250, the disk252 may be formatted at step 305. As discussed above, in order to havethe ability to easy and simply recover from any failures redundancy datamay need to be accurate relative to any corresponding stored data. Inmany cases, this may entail that disks 252 utilized to store a volume beformatted by calculating redundancy data from the current data in theareas on disk 252 where portions of a volume are to be stored, eventhough data stored at these areas may currently be garbage values. Thesecalculations may consume an undesirably large amount of time.

Furthermore, in a distributed RAID environment such as that detailedwith respect to FIG. 1, other problems may present themselves. Morespecifically, as different portions of a volume may be stored ondifferent data banks 110 and redundancy data corresponding to the volumemay also be stored on various data banks 110, accomplishing this type offormatting may additionally require a great deal of communicationbetween distributed RAID applications 210 on data banks 110, consumingprocessor cycles and communication bandwidth.

Thus, in one embodiment, to ensure that redundancy data corresponding toan area of a disk where data of a volume is to be stored is accuraterelative to the area of disk 252 where that data of the volume is to bestored, a zero value may be written to the areas on disks 252 where datacorresponding to the volume is to be stored and the areas on disk 252where redundancy data is to be stored. By zeroing out both the areas ofa disk 252 where data of a volume is to be stored and areas of disks 252where redundancy data is to be stored it can be guaranteed that any dataof the volume can be recreated from its corresponding redundancy data.

Zeroing disks 252 may have other advantages. Namely that no complexcalculations may need to be performed to determine redundancy data andno communications between distributed RAID applications 210 may be toachieve relative accuracy between areas where a volume is to be storedand redundancy data corresponding to those areas.

Importantly, by zeroing out areas of disks 252 for use with a volume andits corresponding redundancy data a significant delay in the usabilityof RAID system 100 may be avoided. These advantages may be attainedthrough the use of a process which substantially continuously duringoperation zeros out unallocated areas of disks 252 resulting, forexample, from the initial use of distributed RAID system 100, theinstallation of new disks 252, the deletion of a volume, etc. In theseinstances, currently unallocated (i.e. not currently allocated) areas ofdisks 252 on each of data banks 110 may have zeros written to them(referred to as “zeroing” the area).

The unallocated areas of disks 252 which have been zeroed may be trackedsuch that when a command corresponding to a portion of a volume orredundancy data associated with a portion of a volume is received at adata bank 110 to which that portion is assigned, distributed RAIDapplication 210 may check to determine if that portion has been assigneda corresponding area of disks 252 on data bank 110 where that portionhas been assigned. If no corresponding area of disks 252 on data bank110 has been assigned, distributed RAID application 210 may select anarea of disks 252 which has been zeroed and assign this area of disks252 to the portion of the volume or corresponding redundancy data.

By simultaneously zeroing out any unassigned areas which have notpreviously been zeroed and waiting until a command corresponding to aportion of a volume or redundancy data is received to assign a zeroedarea of disks 252 to that portion distributed RAID system 100 mayoperate substantially immediately without a long involved formattingprocess and new disks 252 may be added and volumes deleted or freedrelatively unobtrusively to the operation of distributed RAID system100.

It will be noted, therefore, after reading the above that step 305 inwhich the disks 252 are formatted may be accomplished before, during orafter the creation of a volume with respect to distributed RAID system100 and that the placement of step 305 (and all other steps in all theflow diagrams herein) implies no order to the steps. As will also benoted after a thorough review of the rest of the steps in FIG. 3 and theremainder of the disclosure, a volume may be created and portions of thevolume and redundancy data corresponding to the volume assigned to adata bank 110 before physical areas on disks 252 on these data banks 110have been assigned to store the portions of the volume or redundancydata and that, furthermore, the zeroing of the physical areas on disks252 used to store the data corresponding to such portions may occurbefore the creation of the volume or after the creation of the volumebut before these physical areas are assigned to corresponding portionsof the volume or redundancy data (as discussed in more detail later).

These locations may be better explained with reference to the concept ofa segment which may be utilized by embodiments of a distributed RAIDapplication 210, where a segment may be the size of 2048 logical blockaddresses (LBAs) (or some other size) and the size of the logical blockaddress corresponds to the sector size of a disk 252. Disks 252 in thedata store 250 on each of data banks 110 may therefore be separated intoequal size segments (for example, 1 MB) at step 310. These segments maycorrespond to one or more contiguous data blocks of a disk drive 252.Therefore, when a user or host 102 requests the creation of a volumefrom distributed RAID application 210 at step 320 and specifies a levelof RAID which will be used in conjunction with that volume at step 330,a number of these segments corresponding to the requested size of thevolume plus the number of segments desired to implement the desiredlevel of RAID in conjunction with the volume may be assigned to thevolume at step 340.

Thus, the volume comprises a number of segments (also referred to aslogical segments), where each of these segments may be associated with aparticular data bank 110 such that the data bank 110 may be assigned tomanage that segment of the volume. This segment may, for example, may bethe size of 2048 logical block addresses (LEAs), where the size of thelogical block address corresponds to the size sector size of a disk 252(other arrangements and sizes will also be possible). In most cases thephysical storage comprising that segment of the volume will be stored inthe data store 250 of the data bank 110 which manages that segment,however, in other cases the data corresponding to that segment may bestored in the data store 205 of a different data bank 110 (in otherwords, in certain cases the data bank 110 comprising the distributedRAID application which manages that segment may be distinct from thedata bank 110 comprising the data store 250 which stores the datacorresponding to that segment).

In one embodiment, the allocation of segments to data banks 110corresponding to a particular volume may be accomplished by determininga random permutation of the set of data banks 110 corresponding to thedistributed RAID system 100. Thus, if there are six data banks a randompermutation of size six, where the random permutation comprises each ofthe data banks may be determined such that the segments may assigned toeach of the data banks consecutively in the order of the randompermutation.

For example, suppose there are four data banks 110 in a distributed RAIDsystem (call them data bank1, data bank2, etc.). A random permutation ofdata bank2, data bank4, data bank1 and data bank3 may be determined. Inthis case, the first segment corresponding to a volume is on data bank2,the second segment may be on data bank4, the third on data bank1, thefourth on data bank 3 and the fifth back again on data bank 4. In thisway, the location of a particular segment corresponding with the volumemay be determined mathematically if the random permutation correspondingto the volume is known.

As mentioned the user may specify that a level of RAID is to beimplemented in conjunction with a volume at step 330. In this case,distributed RAID application 210 may ensure that any data correspondingto the implementation of RAID in conjunction with a volume is stored atan appropriate location at step 350 such that the RAID information isappropriately distributed across data banks 110 to ensure that thedesired level of RAID is achieved.

For example, if it is desired to implement RAID 5 in conjunction with avolume, distributed RAID application 210 may determine a desired RAIDparity group size (for example, based on a user configured RAID set orotherwise determined). This determination may be based on the number ofdata banks 110 in the distributed RAID system and may, in oneembodiment, be one less than the number of data banks 110 (plus anadditional one to account for the parity data).

To illustrate, if there were five data banks 110, for every foursegments which store data associated with the volume (referred to asdata segments), one segment would be dedicated to parity and the parityfor the four segments calculated and stored in this parity segment,where the parity segment would be dedicated in a data bank 110 whosedata store 250 does not comprise the data segments from which the paritydata of the parity segment was calculated.

At this point, each segment corresponding to a logical volume has beenassigned to a particular data bank 110 and any segments 100 to beutilized to store RAID data corresponding to the volume (referred toherein interchangeably as redundancy segments or parity segments,without loss of general applicability to the use of the segment to storeany type of redundancy data associated with the implementation of anylevel of RAID in conjunction with a volume) have also been assigned to adata bank 110, however, physical sectors of the disks 252 of the datastores 250 of the data banks may not have yet been assigned to store thedata corresponding to those segments. Thus, at step 360 physicalsegments of disks 252 on the data bank 110 to which a logical segment ofthe volume has been assigned may be determined and assigned to thelogical segments. This segment mapping may be stored in the local tables245 of each data bank 110. This assignment may, as mentioned earlier,take place at some later point, for example, when a command firstattempts to write a logical segment.

When making this assignment, in one embodiment the areas differentperformance characteristics of disks 252 may be accounted for relativeto the accessed logical segment. In other words, disks 252 may havesegments which are more efficiently accessed than other segments of thesame disk. Therefore, in one embodiment it may desirable to assignphysical segments of a disk 252 based upon criteria associated with thelogical segment. The characteristics may include for example, suchthings as a quality of service designation associated with a volumecorresponding to the logical segment, a number of accesses to the volumecomprising the logical segment, etc.

At step 370, then, information corresponding to the volume may bestored, such that the location of segments corresponding to the volume,or segment corresponding to the implementation of RAID in conjunctionwith the volume, may be determined from this stored information. Thisstored information (collectively referred to as mapping data) maytherefore include an identification for the volume, the randompermutation corresponding to the volume (for example, indicating theorder of data banks 110 on which the segments are located) and theparity group size of any RAID implementation (for example, if the volumecorresponds to a 4+1 RAID set, a 7+1 RAID set, if RAID 1 is implemented,etc.). This data may be stored, for example, in global tables 240 suchthat it can be communicated to other distributed RAID applications 210on other data banks 110 to ensure that at least a portion of the set oftables 240 associated with each distributed RAID application 210 remainssubstantially consistent and the location.

The above description may be better understood with reference to FIG. 4which depicts one example of one embodiment of a distributed RAID systemwith five data banks 110. Here, each data store 250 of each data bank110 has been laid out as a set of equally sized segments 402, which forpurposes of this example will be assumed to be 1 MB in size. Suppose,now that a host 102 requests a volume of 8 MB with RAID level 5 fromdistributed RAID application 210 b on data bank 110 b. In this case,distributed RAID application 210 b may determine that eight segments 402are required for the data portion of the volume while two segments 402segments may be required to store the redundancy data for the volume inconjunction with implementation of the desired RAID level for thevolume. Distributed RAID application 210 b may then determine a randompermutation for the volume. For purposes of this example, assume thatthe random permutation is: data bank 110 b, data bank 110 d, data bank110 a, data bank 110 c, and data bank 110 e. Thus, data bank 110 b maybe assigned segment 402 a, the first segment of the requested volume,data bank 110 d may be assigned segment 402 b, the second segment of therequested volume, data bank 110 a may be assigned segment 402 c, thethird segment of the requested volume, data bank 110 c may be assignedsegment 402 d, the fourth segment of the requested volume and data bank110 e may be assigned segment 402 e, the fifth segment of the requestedvolume. The assignment then begins again with the first data bank 110 ofthe random order such that data bank 110 b may be assigned segment 402f, the sixth segment of the requested volume, data bank 110 d may beassigned segment 402 g, the sixth segment of the requested volume anddata bank 110 a may be assigned segment 402 h, the eighth segment of therequested volume.

Once the data segments 402 a-402 h for storing data associated with thevolume have been assigned, distributed RAID application 210 b may assignsegments 402 for any data associated with the implementation of thatRAID level. In this example, as RAID 5 is desired with respect to thevolume, distributed RAID application 210 b may determine that as fivedata banks 110 are being utilized a (4+1) parity set may be desired.Distributed RAID application 210 b may then determine that to store theparity to implement RAID 5 in conjunction with eight segments 402 anadditional two segments 402 may be needed.

Furthermore, it may be desired that the parity created utilizing aparticular set of data segments 402 will not be stored on a data bank110 having any of those set of data segments 402 in its data store.Thus, distributed RAID application 210 b may also determine a locationwhere each of the parity segments will be allocated based on thedetermined RAID parity group size, the location of the first datasegment 402 a, etc. Here, parity segment 402 i which will store theparity data corresponding to the data stored in data segments 402 a, 402b, 402 c and 402 d will be allocated in data store 250 c of data bank110 c while parity segment 402 j which will store the parity datacorresponding to the data stored in data segments 402 e, 402 f, 402 gand 402 h will be allocated in data store 250 e of data bank 110 e.Notice here that the parity segments 402 i, 402 j which will store theparity information associated with the implementation of RAID inconjunction with the volume comprising data segments 402 a-402 h arelaid out and sized substantially identically to as those segments 402a-402 h which store the data associated with the volume.

Thus, when a host 102 accesses the volume, a request with a logicaladdress corresponding to the first data segment of the volume maycorrespond to data segment 402 a on data bank 110 b, a request with alogical address corresponding to the second data segment of the volumemay correspond to data segment 402 b on data bank 110 d, etc. Noticehere, that the allocated data segments 402 a-402 h may reside ondifferent data banks 110 and that the location of any allocated datasegment 402 a-402 h may be determined using the random permutationassociated with that volume (for example, as stored in global tables 240at data banks 110). As discussed above, however, data stores 250 on databanks 110 have been virtualized, thus the requesting host may not beaware of the location of the data segments 402 in data stores 250, thatmultiple data stores 250 exist, that data stores 250 are spread acrossmultiple data banks 110, etc. Host 102 believes it is addressing asingle contiguous volume.

It will be apparent that the location of the data segments 402 on databanks 110 (and the corresponding random permutation of data banks 110)in this example is for purposes of illustration and that the datasegments 402 of a volume may be located on any of data stores 250 on anyof the data banks 110 according to almost any random, or other,permutation. Furthermore, it will be noted that while each of segments402 is in this example 1 MB, these may be of any size without loss ofgenerality and that a 1 MB size has been chosen solely for ease ofillustration.

As can be seen from the above description then, the location of aparticular data segment 402 or parity segment 402 can be determinedalgorithmically (for example, using the same random permutation used toassign segments for the volume, locate the parity segments for thevolume, etc.) using the random permutation associated with the volumeand the RAID parity group size. Thus, the information may be stored inconjunction with an identification corresponding to the volume, forexample in set of global tables 240. Furthermore, these global tables240 may be communicated between data banks 110, or otherwise updated,such that at least portions of the set of global tables 240 in each ofthe data banks 110 may be kept substantially consistent.

It may be helpful here to briefly delve into more detail regardingglobal tables 240 associated with distributed RAID application 210. Asdiscussed, in one embodiment, global tables 240 may store informationassociated with volumes created by distributed RAID application 210where those tables 240 can be used to determine a data bank 110associated with a data segment within that volume or where a paritysegment associated with a data segment corresponding to that volume islocated. Global tables 240 may therefore comprise a set of tables, eachtable corresponding to a volume implemented with respect to databanks110. In particular, one of these tables 240 may contain data which maybe used to identify a data bank 110 whose data store 250 comprises acertain segment of a volume. Specifically, this table may be used tocorrelate a logical address associated with a volume with the data bank110 where the segment (data, redundancy, etc.) corresponding to thatlogical address is stored.

FIG. 5 depicts a graphical representation of one embodiment of this typeof table, where each volume may have an associated instance of such atable associated. Table 550 includes entries for LV number 504, segmentsize 508, segment count 512, quality of service (QOS) 514, range count518, information for range entries, including in the embodiment depicteda first range 524 a and a second range 524 b and any additional rangeentries 524 n.

LV number 504 is a unique number used to identify a particular volume,segment size 508 corresponds to the size of the segments used toimplement the volume, segment count 512 corresponds to the number ofsegments corresponding to the logical volume (for example, both thenumber of data segments and redundancy segments, just the number of datasegments, etc), QOS 514 indicates the quality of service which it isdesired to implement with respect to the volume (note that this QOSindicator may indicate a priority to be given to that volume relative toother volumes stored on data banks 110) and range count 518 indicates anumber of ranges associated with the volume, while range entries 524each correspond to one of those ranges.

A range may correspond to a particular data bank 110 order and RAIDimplementation. Multiple ranges may be utilized to implement a volumefor a variety of reasons. Specifically, for example, multiple ranges maybe utilized in conjunction with a volume because different data stores250 at different data banks 110 may have different amounts of storage indata store 250 available for use. This may lead to a situation where forexample, for a first range of a volume all data banks 110 may beutilized in conjunction with a first RAID implementation while in asecond range of a volume fewer than all the data banks 110 available maybe utilized in conjunction with a second RAID implementation (where thefirst and second RAID implementations may, in fact, be different levelsthan one another). Each of these ranges may therefore correspond tosegments laid out according to different data bank 110 orders (forexample, random permutations, etc.), having a different number of databanks 110 available for use, a different type of RAID, etc.

To illustrate using a concrete example, brief reference is made back toFIG. 4. Suppose that the volume of 8 MB with RAID level 5 is laid out asshown, where the data segments are laid out according to the order databank 110 b, data bank 110 d, data bank 110 a, data bank 110 c, and databank 110 e and RAID 5 is implemented in conjunction with the volumeutilizing a (4+1) parity set may be desired with the parity segmentsassigned in data store 250 c of data bank 110 c and data store 250 e ofdata bank 110 e.

Now suppose that it is requested to add an additional 3 MB to thisvolume. However, suppose in this instance that data stores 250 of databanks 110 e, 110 c and 110 d have no more room. Thus, in this case theonly solution may be to allocate the additional desired 3 MB betweendata banks 110 a and 110 b which have remaining storage in data stores250. Furthermore, as only two data banks 110 may be available for use itmay only be possible to utilize a RAID level of 1 instead of RAID 5 asutilized with the first 8 MB of the volume. Thus, in this case the first8 MB of the volume may correspond to a first range, and have a firstrange entry in a table corresponding to the volume with a first set ofvalues while the next 3 MB of the volume may correspond to a secondrange, and have a second range entry in a table corresponding to thevolume with a second set of values. As may be apparent after readingthis disclosure, this type of occurrence may occur with some frequency.

Returning to FIG. 5, to deal with these types of situations, amongothers, each range of a volume may have an entry in a table 550 suchthat the location of segments in that particular range may be determinedfrom the range entry corresponding to that range. Entries 524 for eachof the ranges of the volume corresponding to the table 550 areassociated with range count 518. In one embodiment, range count 518 maycorrespond to the number of ranges of a volume such that the number ofrange entries 524 corresponds to the range count 518. While only rangeentries 524 a and 524 b are shown it will be noted that the number ofrange entries 524 in a table will depend on the number of rangescorresponding to the volume to which that table corresponds. Thus, if avolume is divided into three ranges, there will be three range entries524 in table 550 such that there is a range entry 524 comprisinginformation for each range of the volume corresponding to table 550.

Information for a range entry 524 includes type 526, start 530, end 534,network RAID 538, network RAID size 542, disk RAID 546, disk RAID size550, databank count 554, databank order 558 and a disk count 562 anddisk order 566 corresponding to each data bank 110 used to storesegments associated with range 524 (in other words there will be a diskcount 562 and disk order 566 equal to databank count 554 of that rangeentry 524). Type 526 describes the type of the range corresponding toinformation for range entry 524: for example, normal, source (SRC),destination (DST) or other type of range. Start 230 is the first logicalsegment address of the range of the volume corresponding to range entry524. End 234 is the last logical segment address of the rangecorresponding to information for the range of the volume correspondingto range entry 524. Other arrangements are also possible, for example,end 524 may be a count which is the maximum number of segments or blocksin the range, etc.

Databank count 562 may correspond to the number of data banks 110 onwhich the range corresponding to the range entry resides, databank order558 may be the order in which segments in that range were assigned todata banks 110 while network RAID 538, network RAID size 542, disk RAID546 and disk RAID size 552 may correspond to the type of RAIDimplemented in conjunction with the range of the volume corresponding torange entry 524.

Network RAID 538 is the type of RAID being implemented in associationwith the volume corresponding to the table 550, for example, RAID 0,RAID 1 or RAID 5 or other RAID types. Network RAID Size 542 is theparity group size of the RAID type used in the range. The Network RAIDSize 542 may be limited by the number of data banks 110 in the range tobe less than or equal to the number of databanks in the rangecorresponding to information for range 524. Disk RAID 546 is the type ofRAID being implemented across disks in the databanks in the range. DiskRAID size 552 may be the parity group size of the RAID type used acrossthe disks 252 in the data store 250 of each data bank 110 and may belimited to be less than or equal to the number of disks in the databank.In embodiments, RAID across the disks in the databanks 110 in the rangeis optional and may or may not be used. In such embodiments, either DiskRAID 546, Disk RAID Size 552 or both may not be used or may be omitted.

Data bank count 554 is the number of databanks in the range and Databankorder 558 is the order in which RAID is implemented (for example,striped) across the data banks 110 in the range. For example, data banks110 may have data corresponding to the logical addresses of the volumesaved in a certain order and databank order 558 corresponds to thisorder. Disk count 562 is the number of disks within a data bank 110 ofthe range and disk order 566 is the order in which RAID is implementedacross disks of a particular databank 110. For example, disks 252 mayhave segments saved to them in a certain order and disk order 566 is theorder in which segments are stored across disks 252 in a data bank 110.Thus, for each databank 110 used to store segments of the rangeassociated with the range entry 524 there will be a corresponding diskcount 562 and disk order 566 (in other words the number of disk counts562 and disk orders 566 will, in one embodiment, be equal to databankcount 554 of that range entry 524). In embodiments, RAID across disks252 in the data banks 110 is optional and may not be used. It will benoted that while table 550 has been described with specificity, thisdescription is by way of example, not limitation and other forms oftable 550 may be utilized. For example, a virtual table may be usedinstead of table 550 and may explicitly list the segment 402 and databank 110 corresponding to each logical address.

Thus, as discussed earlier, information in table 550 may be used toidentify a data bank 110 comprising a data segment 402 corresponding toa logical address (referenced by a host 102 in a command or in any othercontext). For example, knowing the size of segments 402 and using start530, end 534, the range entry 524 corresponding to the address, etc.,the particular data bank 110 corresponding to a logical address of thevolume can be determined.

While one or more portions of tables 240 may be substantially identicalacross all data banks 110 and may describe one or more logical volumeswhich span one or more data banks 110 as described above, other tables245 on a data bank 110 may be distinct to the data bank 110 to which itcorresponds (for instance, table 245 may be unique to the data bank 110on which the corresponding distributed RAID application 210 isexecuting). This table 245 may comprise data pertaining to each disk 252contained in the data store 250 of the corresponding data bank 110 andmay comprise information on where information is stored on or amongdisks 252 of the data store, for example, the sector of a disk 252 wherea segment 402 assigned to the data bank 110 is located in data store250.

In FIG. 6 a graphical representation of one embodiment of this type oftable is depicted. Table 660 may be stored at a particular data bank 110and comprise multiple disk tables 670, each of the disk tables 670corresponding to a disk 252 of the data store 250 within that data bank110 and listing the location of the segments stored within that disk252. More specifically, in most cases disks 252 are divided intophysical sectors, each physical sector having a corresponding address orrange of addresses.

A disk table 670 may be a mapping table which can be utilized todetermine the location of a sector of a disk 252 of the data bank 110where a segment of a volume is stored. Thus, using a table 670 theaddress of a sector on a disk 252 corresponding to a segment of a volumecan be determined. Furthermore, the table may contain one or more flagsor descriptive bits per entry corresponding to a segment or sector ofthe disk, describing the sector or segment stored at that sector.

Referring now to FIG. 7, a graphical representation of one embodiment ofa disk table 670 is depicted. Disk table 670 has multiple entries, eachentry corresponding to a physical segment of the corresponding disk suchthat the entries of disk table 670 describe the physical segments of thedisk 252. Each entry in disk table 670 may also include one or moreflags or bit fields describing the physical segment or segment of thevolume stored at the corresponding sector. More particularly, as shownin FIG. 7, in one embodiment entries in disk table 670 include fieldsfor a logical volume (LV) number, logical segment number, address spaceand sector state. LV number identifies the logical volume to which datastored at that physical segment corresponds. Logical segment numberidentifies the segment of the logical volume corresponding to that data.Address space identifies the segment stored as ‘data’ or ‘redundancy’. Avalue of ‘data’ may indicates that data is stored at the sectorrepresented by the entry, whereas a value of ‘redundancy’ indicates thatthe information stored at the sector may be used for RAID dataprotection and, depending upon the RAID level, may be redundant data,mirrored data or parity information. Sector state indicates the state ofthe segment as being ‘allocated’, ‘zeroed’ or ‘dirty’. ‘Allocated’indicates the segment has been allocated and may comprise valid data.‘Zeroed’ indicates the segment has been zeroed out by writing zeros tothe segment and ‘dirty’ indicates the segment may comprise garbage areotherwise unusable or undesirable values, for example because thesegment has not been zeroed out or allocated, may be storing random bitsor data. In one embodiment, for example, for a new disk all segments ofthe disk may be marked as dirty in a disk table corresponding to the newor newly added disk.

After reading the above description of the tables it will be apparentthat distributed RAID application 210 may utilize the global tables 240to determine which segment corresponds to a logical address of a volume,on which data bank 110 segments corresponding to a volume (either dataor redundancy segments) are located, which segment of a volumecorresponds to a logical address of a volume, where RAID data (paritydata, mirror data, other types of redundancy data, etc.) associated witha segment of a volume is located, which disk 252 on a particulardatabank 110 comprises a segment or other information regarding volumes,segments, or disks 252 corresponding to that particular data bank 110 orother information regarding volumes, segments 402, data banks 110, RAIDdata, etc.

Similarly, distributed RAID application 210 on each individual data bank110 may use local tables 245 on that data bank 110 to determine where onthat data bank 110 (which sector(s) of disk 252, etc.) a particularsegment is located or other information regarding volumes, segments, ordisks 252 corresponding to that particular data bank 110.

Using the combination of the global table 240 shared between data banks110 and the local tables 245 corresponding to each individual data bank110 then, certain operations may be performed by the distributed RAIDapplications 210 on data banks 110 in cooperation with one another.These types of operations will now be discussed in more detail.Specifically, one embodiment of the implementation of a READ command anda WRITE command on a volume where RAID level 5 has been implemented inconjunction with the volume will now be discussed in more detailfollowed by concrete examples of the implementation of these commandswith respect to an example distributed RAID system. It will be noted howother types of embodiments, commands, RAID levels, etc. may beimplemented after a thorough review of this disclosure.

Looking first at FIG. 8, a flow diagram for one embodiment of a methodfor implementing a READ command in a distributed RAID system isdepicted. This READ command may be sent by a host 102 to a data bank 110through a switch 120 or from one data bank 110 to another data bank 110.In certain embodiments, host 102 may comprise one or more applicationsand associated routing information such that a READ command may berouted from the host 102 issuing the command to an appropriate data bank110 along a path between the issuing host 102 and the appropriate databank 110. In other cases, however, no such application or routinginformation may be present on host 102 and thus a READ command issuedfrom a host 102 may be routed to any of data banks 110. It is the lattercase that will be illustrated in this embodiment. After reviewing thedescription of this embodiment, however, it will be noted by those ofskill in the art which steps are applicable to the former case as well.

At step 810, then, a READ command may be received at a data bank 110.The distributed RAID application 210 on data bank 110 may determine, atstep 820, a segment of a volume which corresponds to a logical addressreferenced in the received READ command and on which data bank 110 thesegment of the volume is stored at step 830. As discussed above, thisinformation may be determined using the global tables 240 associatedwith the distributed RAID application 210. If the data bank 110 which isstoring the segment is the same as the data bank 110 which received theREAD command (as determined at step 832) the requested data can beobtained from the appropriate disk 252 of the data store 250 on thereceiving data bank 110 at step 840 and at step 850 the READ commandresponded to. As discussed above, the particular disk 252 of a datastore 250 of the data bank 110 on which a segment is stored can bedetermined using global tables 240 while the location on that disk 252where the data corresponding to the segment is stored may be determinedusing local tables 245 which may be used to map a segment of a volume toa physical location on a disk 252. If the receiving data bank 110received the READ command from the host 102 the host 102 may beresponded to while if the receiving data bank 110 received the READcommand from another data bank 110 the response may be sent to thedistributed RAID application 210 on the data bank 110 which issued theREAD command.

If, however, the segment is stored on a remote data bank 110 (a databank 110 other than the one which received the command) at step 860 theREAD command may be sent to the distributed RAID application 210 at theremote data bank 110. In one embodiment, this READ command may becommunicated to the distributed RAID application 210 at the remote databank 110 using a command format utilized by distributed RAID application210. This command, while providing pertinent information of the originalREAD command may also instruct the distributed RAID application toreturn the result of the READ command to the data bank 110 whichoriginally received that READ command, or to perform otherfunctionality. Accordingly, after the READ command is sent to the remotedata bank 110 at step 870 a response comprising the requested data maybe received from the remote data bank 110 and at step 880 the receivedREAD command responded to using the data received in that response.

Moving, now to FIGS. 9A and 9B, a flow diagram for one embodiment of amethod for implementing a WRITE command in a distributed RAID system isdepicted. This WRITE command may be sent by a host 102 to a data bank110 through a switch 120 or from one data bank 110 to another data bank110. In certain embodiments, host 102 may comprise one or moreapplications and associated routing information such that a WRITEcommand may be routed from the host 102 issuing the command to anappropriate data bank 110 along a path between the issuing host 102 andan appropriate data bank 110. In other cases, however, no suchapplication or routing information may be present on host 102 and thus aWRITE command issued from a host 102 may be routed to any of data banks110. It is the latter case that will be illustrated in this embodiment.After reviewing the description of this embodiment, however, it will benoted by those of skill in the art which steps are applicable to theformer case as well.

At step 910, then, a WRITE command may be received at a receiving databank 110. The distributed RAID application 210 on receiving data bank110 may then determine at steps 920, 930 and 940 the segment of thevolume corresponding to a logical address referenced by the WRITEcommand, the location of that segment (for example, which data banks 110is storing the data corresponding to that segment) and the location ofthe parity corresponding to that segment (for example, which data bank110 is storing the segment where parity data created from the datacorresponding to that segment is stored). As discussed above, thelocation of both the data segment and the parity segment may bedetermined using global tables 240 stored on the receiving data bank110.

If neither the data segment (the segment storing the data) nor theredundancy segment (in other words, where the parity or other type ofredundancy data created from the data segment) is stored on thereceiving data bank 110 (as determined at steps 950 and 960) the WRITEcommand may be communicated to the distributed RAID application 210 onthe remote data bank 110 on which the data segment is stored at step 964and to the distributed RAID application 210 on the remote parity databank 110 on which the parity segment is stored at step 966. In oneembodiment, this WRITE command may be communicated to the distributedRAID applications 210 at the remote data bank 110 and the remote paritydata bank 110 using a command format utilized by distributed RAIDapplications 210. This command, while providing pertinent information ofthe original WRITE command may also instruct a distributed RAIDapplication 210 to perform other desired functionality.

Accordingly, after the WRITE command is sent to the remote data bank 110and the remote parity data bank completion notifications may be receivedfrom the distributed RAID applications 210 on the remote data bank 110and the remote parity data bank 110 at steps 968 and 970. Once theseacknowledgments are received the WRITE command may be responded to bythe distributed RAID application 210 on the receiving data bank 110.

Returning to step 950, if, however, the data segment is stored at thereceiving data bank 110, it may be determined if the WRITE command wasreceived from a host 102 or another data bank 110 at step 952. If theWRITE command was received from a host 102 the WRITE command may becommunicated to the distributed RAID application 210 on the remoteparity data bank 110 at step 976 and placed in the write cache of thereceiving data bank 110 at step 974. After receiving a completionnotification from the distributed RAID applications 210 on the remoteparity data bank 110 at step 978, the WRITE command may be responded toby the distributed RAID application 210 on the receiving data bank 110at step 980 (for example, a response sent to the host 102). Furthermore,the WRITE command itself may be processed at step 982. This process mayentail the storing of data associated with the WRITE command to the datasegment stored on the receiving data bank 110 or other functionality.

On the other hand, if the WRITE command was not received from a host atstep 952 this may indicate that the WRITE command was received fromanother data bank 110 (which, in many cases, may have been the data bank110 which originally received the WRITE command from a host 102). Inthis case, the data bank 110 may place the received WRITE command in itswrite cache at step 984 and sends a completion notification to theissuing data bank 110 at step 986. At some later point then, the WRITEcommand itself may be processed at step 988.

Returning again to step 950, if the data segment is not stored at thereceiving data bank 110 but the parity segment is stored at thereceiving data bank 110, as determined at step 960, it may be determinedif the WRITE command was received from a host 102 or another data bank110 at step 962. If the WRITE command was received from a host 102 theWRITE command may be communicated to the distributed RAID application210 on the remote data bank 110 where the data segment corresponding tothe WRITE is stored at step 1002 and placed in the write cache of thereceiving data bank 110 at step 1000. After receiving a completionnotification from the distributed RAID applications 210 on the remotedata bank 110 at step 1004 the WRITE command may be responded to by thedistributed RAID application 210 on the receiving data bank 110 at step1006 and the write command processed at step 1008 by the receiving databank 110.

Here, processing the write command may entail that the parity segmentstored at the receiving data bank 110 may be updated based upon thewrite command. This update of the parity segment may be accomplished ina variety of ways, not all of which will be elaborated on herein butwhich will be known to those of ordinary skill in the art. For example,distributed RAID application 210 on parity data bank 110 may perform abacked out write in order to update the parity segment. Performing thisbacked out write may entail obtaining data segments from which theparity segment and performing logical operations (such as exclusive OR(XOR) operations) using the obtained data segments and the data to bewritten associated with the WRITE command. Alternatively, if distributedRAID application 210 on receiving data bank 110 has multiple WRITEcommands corresponding to each of the data segments from which theparity segment was created, a new parity segment may be calculated andthe original parity segment may be updated by replacing it with thenewly calculated parity segment. Other methods for updating the paritysegment may be realized from a review of the disclosures herein and theparticular method utilized to update a parity segment by a distributedRAID application may depend on a variety of factors, includingconfiguration parameters, the availability of certain data (for example,WRITE commands corresponding to all data segments used to create theparity, etc.) or any of a number of other factors.

Returning now to step 962, if the WRITE command was not received from ahost this may indicate that the WRITE command was received from anotherdata bank 110 (which, in many cases, may have been the data bank 110which originally received the WRITE command from a host 102). In thiscase, the WRITE command may be placed in the write cache of thereceiving data bank 110 at step 990 and a completion notification sentto the issuing data bank at step 992. The WRITE command may then beprocessed at step 994 (for example, the parity segment may be updated asdiscussed above).

After reviewing the above discussion it will be noted that in manycases, a distributed RAID application 210 at a particular data bank 110may not be able to process a received WRITE command until notificationis received from a parity data bank 110, that a parity data bank mayneed to evaluate multiple received WRITE commands to determine orimplement a method for updating the parity or any of a number of otherinstances when it may be desired to store one or more WRITE commands orevaluate a set of these stored WRITE commands. To facilitate the storageand evaluation of WRITE (or other) commands, each distributed RAIDapplication 210 may have an associated write cache 260.

A representation of one embodiment of a write cache is depicted in FIG.10. As WRITE commands are received by distributed RAID application 210they are placed in write cache 1100. Each of these WRITE commands 1160may have an associated timestamp indicating when the WRITE command 1160was received. Thus, in one embodiment write cache 1100 may comprise aqueue of time stamped WRITE commands 1160. At some point a timestampmarker may be issued by distributed RAID application 210. This timestampmarker may comprise a time and be communicated to each of distributedRAID applications 210. When to issue a timestamp market may bedetermined in a variety of ways, such as when the write cache 1100 is acertain percentage full or when a certain number of WRITE commands 1160have been received, at a certain time interval or a variety of othermethodologies.

In any event, this timestamp marker 1110 will segment each of the writecaches 1110 associated with each of the distributed RAID applications210 into at least two segments a closed marker 1150 comprising WRITEcommands 1160 received before the timestamp marker 1110 (in this exampleWRITE commands 1160 a, 1160 b, 1160 c and 1160 d) and an open marker1140 comprising WRITE commands 1160 received after the timestamp marker1110 (in this example WRITE commands 1160 e, 1160 f and 1160 g).Distributed RAID application 210 may then evaluate the set of WRITEcommands 1160 in the closed marker 1150 (in this example WRITE commands1160 a, 1160 b, 1160 c and 1160 d) to determine how these WRITE commands1160 are to be processed while received WRITE commands may still beadded to open marker 1140.

Conversely, as the closed marker 1150 comprises a set of WRITE commandswhich are no longer changing distributed RAID application may evaluatethis set of WRITE commands 1160 with respect to one another (or othercriteria) to determine an order of execution (and may therefore reorderWRITE commands 1160 in closed marker 1160), a methodology to update aparity segment (for example, if there are WRITE commands in closedmarker 1150 which correspond to each data segment used to create aparity) or make other determinations associated with the processing ofWRITE commands 1160. It will be noted that as a timestamp marker 1110may be issued for multiple reasons by any of distributed RAIDapplications 210 on each of data banks 110, multiple closed markers mayexist at any one point, for example, when multiple timestamp markers1110 are issued by distributed RAID applications 210 between the timethe write cache is evaluated by any one of the distributed RAIDapplications 210.

After reviewing the above the reader may now have an understanding ofhow distributed RAID applications 210 on data banks 110 operate intandem to achieve virtualized storage and RAID implementation. It may befurther helpful to an understanding to certain embodiments, however, todiscuss the functioning of certain embodiments of distributed RAIDapplication 210 after the occurrence of a fault. As discussed above,distributed RAID application 210 may be aware (for example, have stored)of a data bank which is faulty (in other words, which may have ahardware, software, communication or other fault which impedes orhampers the ability of the data bank 110 to operate or access data).Distributed RAID application 210 may be able to account for such faultswhile satisfying commands from hosts 102.

To illustrate, FIG. 11 depicts a flow diagram for one embodiment of amethod for implementing a READ command in a distributed RAID system. Itwill be understood that this method applies to an embodiment where RAID5 has been implemented in conjunction with a range of a volume and thatother embodiments may be equally well applied in cases where otherlevels (or no level) of RAID have been implemented. The READ command maybe sent by a host 102 to a data bank 110 through a switch 120 or fromanother data bank 110 to the receiving data bank 110. At step 1110,then, a READ command may be received at a data bank 110. The distributedRAID application 210 on data bank 110 may determine, at step 1220, adata segment which corresponds to a logical address referenced in thereceived READ command and on which data bank 110 the data segment isstored at step 1230. If the data bank 110 which is storing the datasegment is the same as the data bank 110 which received the READ command(as determined at step 1240) the requested data can be obtained from theappropriate disk 252 of the data store 250 on the receiving data bank110 at step 1250 and the received READ command responded to using theobtained data at step 1252. If the receiving data bank 110 received theREAD command from the host 102 the host 102 may be responded to while ifthe receiving data bank 110 received the READ command from another databank 110 the response may be sent to the distributed RAID application210 on the data bank 110 which issued the READ command.

If, however, the data segment is stored on a remote data bank 110 (adata bank 110 other than the one which received the command) at step1254 it may be determined if the remote data bank 110 on which the datasegment to be read is stored has experienced a fault. If not the READcommand may be sent to the distributed RAID application at the remotedata bank 110. After a response comprising the requested data isreceived from the remote data bank 110 at step 1258 the received READcommand may be responded to using that data at step 1260.

If the remote data bank 110 has experienced a fault, however, it may bedetermined at step 1254 if the receiving data bank 110 holds the paritysegment corresponding to the data segment associated with the READcommand. If the parity segment is stored at the receiving data bank 110the data segment corresponding to the READ command may be obtained usingthe parity segment stored at the receiving data bank 110. Obtaining thedata segment from the parity data may be accomplished in a variety ofway which will not be elaborated on in more detail, including obtainingthe other data segments (data segments other than the one correspondingto the READ command) from other data banks 110 and obtaining the desireddata segment by performing logical operations between the other datasegment and the parity segments. Once the requested data segment hasbeen obtained using the parity data at step 1268 the received READcommand may be responded to at step 1270. If the receiving data bank 110received the READ command from the host 102 the host 102 may beresponded to while if the receiving data bank 110 received the READcommand from another data bank 110 the response may be sent to thedistributed RAID application 210 on the data bank 110 which issued theREAD command.

If the receiving data bank 110 is not the data bank 110 storing theparity block a READ command may be sent to the remote data bank 110 onwhich the parity segment corresponding to the data segment referenced inthe READ command is stored at step 1262. After a response comprising therequested data is received from the remote parity data bank 110 at step1264 the received READ command may be responded to using that data atstep 1266.

Moving on to FIG. 12, a flow diagram for one embodiment of a method forimplementing a WRITE command in a distributed RAID system is depicted.This WRITE command may be sent by a host 102 to a data bank 110 througha switch 120 or from one data bank 110 to another data bank 110. At step1310, then, a WRITE command may be received at a receiving data bank110. The distributed RAID application 210 on receiving data bank 110 maythen determine at steps 1312, 1314 and 1320 the data segmentcorresponding to a logical address referenced by the WRITE command, thelocation of that data segment (for example, which data banks 110 isstoring the data corresponding to that segment) and the location of theparity segment corresponding to that data segment (for example, whichdata bank 110 is stores the segment where parity data created from thedata corresponding to that segment is stored). As discussed above, thelocation of both the data segment and the parity segment may bedetermined using the set of tables 240 stored on the receiving data bank110.

It can then be determined if either the data bank 110 on which the datasegment is stored or the data bank 110 on which the parity segment isstored have experienced a fault. If neither of those data banks 110 hasexperience a fault a normal write operation may be carried out at step1324 by the distributed RAID application. A normal write operation hasbeen discussed previously with respect to FIGS. 9A and 9B and will notbe discussed further.

If however, either of those data banks 110 has experienced a fault awrite operation taking into consideration the failed data bank 110 maybe conducted at step 1326. This write operation may parallelsubstantially identically the write operation described with respect toFIGS. 9A and 93 with the exception that before a write command is sentto a data bank 110 it is determined if the data bank 110 to which thewrite is to be sent is failed and if so that write command is not sentto the failed data bank 110. In all other respects the write operationis substantially identical.

After reviewing the aforementioned flow diagrams the operation ofcertain embodiments may be better understood with reference to specificexamples of one embodiment of a distributed RAID system in operation. Tothat end, attention is directed back to FIG. 4 which illustrates oneembodiment of a distributed RAID system with five data banks 110. Tobegin with a first example, suppose that host 102 b issues a READcommand to data bank 110 c, where the READ command references a logicaladdress which corresponds to data segment “2” 402 c on data bank 110 a.Here, distributed RAID application 210 c on data bank 110 c maydetermine that the logical address of the received READ commandreferences data segment “2” 402 c and that data segment “2” 402 c isstored on data bank 110 a. Distributed RAID application 210 c may thensend a corresponding READ command to data bank 110 a.

Distributed RAID application 210 a on data bank 110 a may receive thisREAD command, determine that the READ command references a logicaladdress which corresponds to data segment “2” 402 c and that datasegment “2” 402 c is located on the data bank 110 a on which it isexecuting. Distributed RAID application 210 a may then access data store250 a to obtain the data requested from data segment “2” 402 c andreturn this obtained data to the distributed RAID application 210 c atissuing data bank 110 c. Distributed RAID application 210 c on data bank110 c may receive this response from distributed RAID application 210 aon data bank 110 a and use data from this response to respond to theoriginal READ command issued from host 102 b.

Now suppose that host 102 b issues a READ command to data bank 110 c,where the READ command references a logical address which corresponds todata segment “2” 402 c on data bank 110 a, but that data bank 110 a hasexperience a fault and is no longer operating. In this case, distributedRAID application 210 c on data bank 110 c may determine that the logicaladdress of the received READ command references data segment “2” 402 cand that data segment “2” 402 c is stored on data bank 110 a.Additionally, distributed RAID application 210 c on data bank 110 c mayalso determine that data bank 110 a has experienced a fault.

Accordingly, distributed RAID application 210 c may determine that thelocation of parity segment 402 j corresponding to data segment “2” 402 cis data bank 110 e. Distributed RAID application 210 c may then send aREAD command to data bank 110 e. Distributed RAID application 210 e ondata bank 110 e may receive this READ command, determine that the READcommand references a logical address which corresponds to data segment“2” 402 c and that the parity segment 402 j corresponding to datasegment “2” 402 c is located on the data bank 110 e on which it isexecuting. Distributed RAID application 210 e may then access data store250 e to access parity segment 402 j and obtain the data requested fromdata segment “2” 402 c using the parity segment 402 j. This obtaineddata may be returned to the distributed RAID application 210 c atissuing data bank 110 c. It will be noted that distributed RAIDapplication 210 e may need other data to determine the data requestedfor data segment “2” 402 c. Accordingly, distributed RAID application210 e may determine that the location of data segment “0” 402 a, datasegment “1” 402 b and data segment “3” 402 d which were used inconjunction with data segment “2” 402 c to create parity segment 402 jare located respectively on data banks 110 b, 110 d and 110 c.Distributed RAID application 210 e may thus obtain data segment “0” 402a, data segment “1” 402 b and data segment “3” 402 d by sending READrequests to these data banks 110 b, 110 d and 110 c and use data segment“0” 402 a, data segment “1” 402 b and data segment “3” 402 d inconjunction with parity segment 402 j to obtain the data requested fromdata segment “2” 402 c.

Distributed RAID application 210 c on data bank 110 c may receive theresponse from distributed RAID application 210 e on data bank 110 e anduse data from this response to respond to the original READ commandissued from host 102 b. In this manner, data corresponding to a datasegment can still be read by a host despite the occurrence of a fault inthe distributed RAID system.

Continuing on with WRITE commands, suppose that host 102 b issues aWRITE command to data bank 110 c, where the WRITE command references alogical address which corresponds to data segment “2” 402 c on data bank110 a. Here, distributed RAID application 210 c on data bank 110 c maydetermine that the logical address of the received WRITE commandreferences data segment “2” 402 c and that data segment “2” 402 c isstored on data bank 110 a. Furthermore, distributed RAID application 210c may determine that the parity segment 402 j corresponding to datasegment “2” 402 c is located on data bank 110 e. Distributed RAIDapplication 210 c may then send a corresponding WRITE command to databanks 110 a and 110 e. Upon receiving completion notifications fromdistributed RAID applications 210 a and 210 e, distributed RAIDapplication 210 c may respond to the originally received WRITE command.

Distributed RAID application 210 e on data bank 110 e may receive itscorresponding WRITE command, determine that the WRITE command referencesa logical address which corresponds to data segment “2” 402 c and thatthe parity segment 402 j corresponding to data segment “2” 402 c islocated on the data bank 110 e on which it is executing. DistributedRAID application 210 e may place the WRITE command in its write cacheand send a completion notification to data bank 110 c. Distributed RAIDapplication 210 e may then access data store 250 e to access paritysegment 402 j and update the parity segment 402 j using the datareferenced in the received WRITE command.

Distributed RAID application 210 a on data bank 110 a may receive itscorresponding WRITE command, determine that the WRITE command referencesa logical address which corresponds to data segment “2” 402 c and thatdata segment “2” 402 c is located on the data bank 110 a on which it isexecuting. Distributed RAID application 210 a may place the WRITEcommand in its write cache and send a completion notification to databank 110 c. Distributed RAID application 210 a may then access datastore 250 a to update the segment “2” 402 c using the data referenced inthe received WRITE command.

Again suppose now that host 102 b issues a WRITE command to data bank110 c, where the WRITE command references a logical address whichcorresponds to data segment “2” 402 c on data bank 110 a, but that databank 110 a has experience a fault and is no longer operating. In thiscase, distributed RAID application 210 c on data bank 110 c maydetermine that the logical address of the received WRITE commandreferences data segment “2” 402 c and that data segment “2” 402 c isstored on data bank 110 a. Additionally, distributed RAID application210 c on data bank 110 c may also determine that data bank 110 a hasexperienced a fault. Furthermore, distributed RAID application 210 c maydetermine that the parity segment 402 j corresponding to data segment“2” 402 c is located on data bank 110 e. Distributed RAID application210 c may then send a corresponding WRITE command to data bank 110 e.Upon receiving a completion notification from distributed RAIDapplications 210 e distributed RAID application 210 c may respond to theoriginally received WRITE command.

Distributed RAID application 210 e on data bank 110 e may receive thecorresponding WRITE command, determine that the WRITE command referencesa logical address which corresponds to data segment “2” 402 c and thatthe parity segment 402 j corresponding to data segment “2” 402 c islocated on the data bank 110 e on which it is executing. DistributedRAID application 210 e may place the WRITE command in its write cacheand send a completion notification to data bank 110 c. Distributed RAIDapplication 210 e may then access data store 250 e to access paritysegment 402 j and update the parity segment 402 j using the datareferenced in the received WRITE command. In this manner, datacorresponding to a data segment can still be written by a host despitethe occurrence of a fault in the distributed RAID system.

After reviewing the above embodiments of distributed RAID systems it maybe observed that during the operation of such distributed RAID systems anumber of eventualities may occur. The eventualities may for example,entail alterations to such a distributed RAID system, such as, forexample, the addition of new storage within a data bank, the failure orremoval of storage within a data bank, the addition or removal of databanks, the alteration of a RAID level or RAID type of a volume, etc.Alterations of these types, or any of a number of other causes, mayentail the movement of data comprising a volume, movement of redundancydata corresponding to a volume, the creation and placement of redundancydata corresponding to a volume, that data comprising a volume orcorresponding redundancy data be created or moved among the plurality ofdata banks or within a single data bank, etc. (collectively referred toherein as migration). It is therefore desired to handle such migrationas seamlessly as possible, such that a user's access to data of thevolume may be minimally affected.

It may be helpful here, before delving into more detail about themigration process, to describe certain instances in which embodiments ofsuch a migration processes may be effectively utilized. FIG. 13 depictsan embodiment of such a scenario where a data bank 110 has been added tothe embodiment of the distributed RAID system of FIG. 4. In FIG. 4, avolume is stored on a distributed RAID system with five data banks 110.The volume comprises a set of equally sized segments 402 a-402 f wherethe random permutation of the data banks 110 used to store data segmentsof the volume is: data bank 110 b, data bank 110 d, data bank 110 a,data bank 110 c, and data bank 110 e. Furthermore corresponding paritysegments 402 i and 402 j stored on databank 110 c and data bank 110 eare utilized to implement RAID level 5 with a (4+1) parity set inconjunction with the volume.

Now, with reference to FIG. 13, suppose data bank 110 f is added to thedistributed RAID system depicted in FIG. 4. When data bank 110 f isadded, the amount of storage in data store 250 f may be likewise addedto the storage available for use by the distributed RAID system. Inorder to effectively integrate data bank 110 f (and the storage providedby data store 250 f) into the distributed RAID system it may be desiredto migrate at least some data which previously was stored on data banks110 a-110 e to data store 250 f of newly added data bank 110 f. However,as an additional data bank 110 f is now available to store data it maybe desired as well to utilize a different parity set in conjunction withone or more volumes of data stored on the distributed RAID system.

Thus, when a new data bank 110 f is added, data associated with one ormore volumes previously stored on data banks 110 a-110 e may be migratedsuch that data associated with these volumes may be stored on a numberof data banks 110 which may include data bank 110 f. This migrationprocess may encompass moving the data segments of the volume from onedata bank 110 to another, the calculation of new parity segmentscorresponding to the volume (for example to implement a new RAID levelor RAID size), or other movement or calculation of data.

In FIG. 13, one example of a migration of the volume described withrespect to FIG. 4 which may occur with the addition of data bank 110 fis depicted. Here, when it is desired to move the volume a distributedRAID application 210 may determine a new configuration for the volume,including a new random permutation (of data banks 110) for the volume,as a new data bank 110 f has been added. For purposes of this example,assume that the new random permutation for the volume is: data bank 110b, data bank 110 d, data bank 110 f, data bank 110 c, data bank 110 aand data bank 110 e. Furthermore, in this example, as RAID 5 is desiredwith respect to the volume, in determining a new configuration for thevolume the distributed RAID application 210 may determine that as sixdata banks 110 are now being utilized a RAID size of a (5+1) parity setmay be desired. Distributed RAID application 210 may then determine thatto store the parity to implement RAID 5 in conjunction with eightsegments 402 an additional two segments 402 may be needed.

Thus, data bank 110 b may be assigned segment 402 a the first segment ofthe requested volume, data bank 110 d may be assigned segment 402 b thesecond segment of the requested volume, data bank 110 f may be assignedsegment 402 c the third segment of the requested volume, data bank 110 cmay be assigned segment 402 d the fourth segment of the requestedvolume, data bank 110 a may be assigned segment 402 e the fifth segmentof the requested volume and data bank 110 e may be assigned segment 402f the sixth segment of the requested volume. The assignment then beginsagain with the first data bank 110 of the random permutation such thatdata bank 110 b may be assigned segment 402 g the seventh segment of thevolume and data bank 110 d may be assigned segment 402 h the eighthsegment of the volume. The data segments 402 can then be moved (from thelocation depicted in FIG. 4) to their newly determined locations(depicted in FIG. 13) if movement is necessary.

Once the data segments 402 a-402 h for storing data associated with thevolume have been assigned, distributed RAID application 210 b may assignsegments 402 for any data associated with the implementation of thatRAID level. As it is desired that the parity created utilizing aparticular set of data segments 402 will not be stored on a data bank110 having any of those set of data segments 402 currently in its datastore 250. Thus, distributed RAID application 210 may also determine alocation where each of the parity segments will be allocated based onthe determined RAID size, the location of the first data segment 402 a,etc. Here, parity segment 402 j which will store the parity datacorresponding to the data stored in data segments 402 a, 402 b, 402 c,402 d and 402 e will be allocated in data store 250 e of data bank 110 ewhile parity segment 402 i which will store the parity datacorresponding to the data stored in data segments 402 f, 402 g and 402 hwill be allocated in data store 250 c of data bank 110 c.

After the assignment of the location of the data segments and redundancysegments for the new configuration of the volume, distributed RAIDapplication 210 may facilitate the migration of the volume by moving thedata segments from their previous location according to the oldconfiguration to the new location assigned according to the newlydetermined configuration if needed. Here, for example, segment “0” 402 amay be in data store 250 b of data bank 110 b in the old configuration(as depicted in FIG. 4) and may be assigned to data bank 110 b in thenew configuration (represented by FIG. 13), thus it may not be necessaryto move segment 402 a. Segment “2” 402 c, however, may be assigned to,and stored in the data store 250 a of, data bank 110 a in the oldconfiguration while in the new configuration of the volume segment “2”402 b is assigned to data bank 110 f, thus it segment “2” 402 b may bestored in data store 250 f of data bank 110 f. In general then, if asegment resides on a data bank 110 in the old configuration and isassigned to the same data bank 110 in the new configuration no movementmay be necessary, while of the data bank on which the segment resides isdifferent from the data bank to which it is assigned in the newconfiguration movement of the segment may occur during the migrationprocess.

The migration process may also entail the creation or movement ofredundancy segments, if needed. Here, in the old configuration of thevolume (represented by FIG. 4) the volume was configured according toRAID size of a (4+1) parity set, thus, segment 402 i stored on data bank110 c and segment 402 j stored on data bank 110 e are created from fourdata segments according to the old configuration. However, in the newconfiguration it is desired to utilize a (5+1) parity set. Thus,distributed RAID application 210 may, during the migration process,calculate data for the parity segment 402 j from data segment “0”, datasegment “1”, data segment “2”, data segment “3” and data segment “4” andstore this calculated in the parity segment 402 j. Similarly,distributed RAID application 210 may calculate data for the paritysegment 402 i from data segment “5”, data segment “6” and data segment“7” and store this calculated in the parity segment 402 i. Note herethat, in this particular example, the parity segments for the newconfiguration have been assigned to the same data banks 110 as in theold configuration. However, this may not be the case in othersituations, and in such cases, if needed the newly calculated paritydata may be stored on a different data bank to which the correspondingparity segments is assigned.

Notice then, with respect to this example, that a volume previouslystored in an old configuration and existing on a distributed RAID systemhaving five data banks 110 has been migrated in conjunction with theaddition of a new data bank 110 to the distributed RAID system such thatthe volume is now stored in a new configuration across the data banks110 of the newly configured distributed RAID system, including the newlyadded data bank 110 f. Additionally notice that the parity set used inconjunction with the implementation of RAID with respect to the volumein the different configurations has also changed based on the additionof data bank 110 f to the distributed RAID system.

Another scenario in which the migration of data in conjunction with adistributed RAID system may be desired is depicted with FIG. 14. Asdescribed above, each data bank 110 has a data store 250 comprising aset of disks 252. As each data store 250 may comprise a different numberof disks 252 and each disk 252 may have different capacity each datastore 250 of each data bank 110 may be of a different size. Thisvariability may lead to situations in which volumes stored in adistributed RAID system may be stored across the data stores 250 offewer data banks 110 (for example, than other volumes or than theaggregate number of data banks 110 in the distributed RAID system) orwhich may have different parity or redundancy sets (assuming, forexample, the case that they have both implemented in conjunction withthe same RAID level). Furthermore, these conditions may exist withrespect to the same volume. In other words, portions of the same volumemay be stored less than all the data banks 110 or portions of the samevolume may have different parity or redundancy sets.

Accordingly, when disks 252 are added to one or more data stores 250 ofdata banks 110 it may be desired to migrate data associated with one ormore volumes such that data associated with these volumes may be storedon the newly added disk 252. By migrating such a volume, a biggerportion of the volume may be stored across more of the data banks 110 ordifferent parity or redundancy set may be implemented in conjunctionwith the volume, which may, in turn, increase the efficiency ofoperation of the distributed RAID system, increase the security of data,decrease access times, or achieve a whole host of other advantages.

With reference to FIG. 14, suppose that data stores 250 a, 250 b, 250 c,250 d comprise 18 terabytes of storage while data store 250 e comprises16 terabytes of storage. Here, volume “1” 1400 may be stored on each ofdata stores 250 a, 250 b, 250 c, 250 d and 250 e. Now suppose thatvolume “2” 1410 is created in the distributed RAID system and that it isdetermined that volume “2” 1410 will be implemented in conjunction withRAID level 5 and a (4+1) RAID parity set. After storing portion 1410 aof volume “2” however, it is determined that all the storage of datastore 250 e on data bank 110 e has been utilized. In this case, thesecond portion of volume “2” 1410 b may be stored only on data stores250 a, 250 b, 250 c and 250 d and may be implemented in conjunction witha (3+1) RAID parity set. Thus, a first portion 1410 a of volume “2” isstored on all data banks 110 and has one type of parity set (4+1) whileanother portion 1410 b of volume “2” is stored on fewer data banks 110and is implemented with a different parity set (3+1). At this point, adisk 252 may be added to data store 250 e increasing the size of datastore 250 e to 18 terabytes. Here it may be desired to migrate portion1410 b of volume “2” such that both portions 1410 a and 1410 b of volume“2” may be stored on the data stores 250 of all data banks 110 and thatboth portions 1410 a, 1410 b may be implemented in conjunction with thesame size parity set (4+1).

It will be apparent from the above discussion that there may be avariety of situations where it is desired to migrate data associatedwith a volume, including the addition or removal of data banks 110, theaddition or removal of storage from one or more data stores 250 on adata bank 110, a change in the level of RAID implemented in conjunctionwith a volume, a failure of one or more disks 252 of a data store 250 orof one or more data banks 110, some combination of the above, or forsome other reason entirely.

Moving to FIG. 15, one embodiment of a method for migrating data in adistributed RAID system is depicted. In response to a trigger, asdiscussed above, one or more volumes may be selected as source volumesat step 1505. A volume may be selected based on a variety of criteria,for example, if a volume comprises one or more portions which are storedon different numbers of data banks 110 or which have different paritysets, that volume may be selected when storage is added. Volumes may beselected as source volumes based on size, on age, on the number of databanks 110 on which they are stored (for example, volumes stored on thefewest number of data banks 110 may be selected first), on RAID level,or on almost any other criteria desirable.

For a selected source volume, then, at step 1510 a range of the sourcevolume is selected. As discussed above with respect to FIG. 5, eachvolume may have one or more ranges, where each range corresponds to anumber of segments of the volume. Once again, then, a variety ofcriteria may be utilized to select a range of a source volume. Thesecriteria include the number of segments in a range, the RAID level orparity size associated with the range, the data bank count associatedwith the range (number of data banks 110 on which that range is stored),or almost any other criteria desirable.

Once a particular range of a source volume is selected it can then bedetermined if there are currently a different number of data banks 110at step 1520. This determination may be made based upon a comparison ofthe number of data banks 110 before the migration process was initiatedand the number of data banks 110 currently installed in the distributedRAID system, or may be made, for example, based upon a comparison of thedatabank count 554 associated with the range and the number of databanks 110 which have room in data store 250 to store the destinationvolume (the source volume after it has been migrated to its newconfiguration) or based upon some other comparison involving the numberor ability (for example, free storage) of data banks 110 to store dataassociated with the corresponding range of the destination volume.

As should be noted, the determination of a different number of databanks 110 may pertain to the number of data banks 110 available to storedata associated with destination volume or range and may not indicatethat a new data bank 110 itself has been added to the distributed RAIDsystem. For example, if storage has been added to a data bank 110already installed in distributed RAID system 110 it may determined thata different number of data banks 110 are available to store dataassociated with the corresponding range of the destination volume, evenif another data bank 110 has not been added to the distributed RAIDsystem (see, for example, FIG. 14 and the accompanying description).

If a different number of data banks 110 is available to store dataassociated with the corresponding range of the destination volume, atstep 1530 a new databank order may be determined for the databank count(which may be, for example, the number of data banks 110 available tostore data associated with the corresponding range of the destinationvolume). In one embodiment, to determine a new databank order a newrandom permutation of the set of data banks 110 available to store dataof the destination volume may be determined. Thus, for example, if adistributed RAID system comprises six data banks and if all six databanks are available to store the corresponding range of the destinationvolume a random permutation of size six, where the random permutationcomprises each of the data banks 110, may be determined such that thesegments of the corresponding range of the destination volume mayassigned to each of the six data banks consecutively in the order of therandom permutation. However, if only five of the six data banks 110 areavailable a random permutation of size five, where the randompermutation comprises each of the five available data banks 110 may bedetermined such that the segments of the corresponding range of thedestination volume may assigned to these five data banks consecutivelyin the order of the random permutation.

After the new databank order for the databank count are established atstep 1530 or if a different number of data banks is not being utilized,a level of RAID to be implemented in conjunction with the correspondingrange of the destination volume and an associated RAID size (forexample, a size of a parity set, etc.) may be determined at step 1540.As mentioned above the user may specify that a level of RAID is to beimplemented in conjunction with a destination volume. If the user hasnot changed the specification of the level of RAID, the new level ofRAID associated with the corresponding range of the destination volumemay be the same as the previous level of RAID associated with the rangeof the source volume.

However, a new level of RAID may also be specified by the user inconjunction with the migration of data of the volume, or it may be thecase that only a certain level of RAID can be implemented because ofcertain restraints associated with the distributed RAID system (forexample, the number of available data banks 110 on which to store dataof the destination volume may dictate that a particular level of RAID beutilized, even if this level of RAID is different than the level of RAIDpreviously implemented in conjunction with the range of the sourcevolume). Similarly, the RAID size may be determined based on the RAIDlevel specified or the number of data banks 110 (for example, databankcount) which will be used to store data associated with the range ordestination volume.

Once the databank order, databank count, RAID level and RAID size forthe new configuration of the range of the volume have been determined(one or more of which may be the same as the existing range of thevolume), the range of the source volume may be partitioned at step 1550.As discussed above, in one embodiment a table may be used to describe avolume, and a range entry in that table may be used to describe a rangeof a volume. Thus, in one embodiment, to partition a range, the rangeentry of the table corresponding to the source volume comprising therange is split into two range entries based upon a migration segmentsize. Additionally, the table describing the volume may be updated toreflect the split of the original range entry into two range entries(for example, by updating the Range Count entry of the table).

More specifically, in one embodiment a new range entry corresponding toa number of segments of the original range equal to the migrationsegment size may be created, where the new range entry describes a rangecomprising a number of segments of the original range (where the numberof segments is equal to the migration segment size). The original rangeentry can be modified such that it no longer describes the segments nowdescribed by the new range entry. In this way two range entries arecreated from the original range entry where the two range entriesdescribe all the segments of the original range entry: specifically anew range entry which describes a range corresponding to a number ofsegments (equal to the migration segment size) of the original rangeentry and a modified original range entry which describes all thesegments of the original range entry except those described by the newrange entry.

It will be noted that except for the designation of which segmentscorrespond to the entry and the type designation of the range entries,the configuration (for example, Databank Count, Databank Order, RAIDlevel, RAID size, etc.) described by the modified original range entryand the newly created range entry are substantially similar at thispoint. The newly created range entry (describing the number of segmentsof the migration segment size) may then be marked as a source forpurposes of the data migration. The marking of this range entry mayentail, for example, changing a Type indicator of the range entry to“SRC” or otherwise to indicate that it is a source range.

The migration segment size may be chosen based on a variety ofvariables, including, in one embodiment, the space reserved to performdata migration in one or more of data stores 250. In particular, in oneembodiment, a portion of each data store 250 on each data bank 110 mayhave space reserved to perform such data migration. The amount of spacemay, for example, be a percentage of the total size of data store 250,may be a fixed amount or may be equal to some other size entirely. Inone embodiment, the amount of reserved space of each data store isequally to approximately 1000 times the segment size.

Thus, the migration segment size may be approximately equal to theamount of reserved space on a data store. So, for example, in oneembodiment 1000 segments of a range may be selected when partitioning arange at step 1550. These segments may, for example be selectedutilizing the addresses of the segments comprising the range such thatthe number of segments starting with the lowest or highest address maybe selected from the segments comprising the original range.

It may be helpful here to give a numerical example, suppose an originalrange comprises 10,000 segments, with the first segment of the rangehaving an address of 500, the last segment having an address of 10,499where the migration segment size is 1000. Here, when the original rangeis partitioned a new range entry describing the segments having theaddresses 500-1,499 may created while the original range entry ismodified to describe the segments of the original range having theaddresses 1,500-10,499. As can be seen then, these two range entriesdescribe all the segments of the original range entry (here, addresses500-10,499). The new range entry (describing addresses 500-1,499) maythen be marked as a source range as described above.

Once the original range has been partitioned to create a source rangeentry, at step 1560 a destination range entry may be created andconfigured. To create a destination range entry the source range entrymay be duplicated and the duplicated range entry marked as a destinationrange entry by, for example, changing a Type indicator of the duplicatedrange entry to “DST” or otherwise to indicate that it is a destinationrange. Thus, there now exist two range entries for the new range entrycreated from the original range entry (discussed above): the sourcerange entry which is a part of the table describing the volume beingmigrated and the destination range entry.

The destination range entry may then be configured by assigning thedestination range entry the values corresponding to the newconfiguration, including the databank count, databank order, RAID leveland RAID size which were determined above for the corresponding range ofthe migrated volume. At this point then, the source range entrydescribes the currently implemented configuration for example, thedatabank count, databank order, RAID level, RAID size, etc.) associatedwith a set of segments (equal in number to the migration segment size)of the original range, while the destination range entry describes thenew configuration associated with the same set of segments as it is tobe implemented in the migrated volume.

Using the source range entry and the destination range entry, the set ofsegments may then be migrated from the configuration described by thesource range entry to the configuration described by the destinationrange entry at step 1570. This migration process may ensure that the setof segments are stored according to the databank order of thedestination range entry and that any data corresponding to theimplementation of RAID in conjunction with the destination range (inother words according to the RAID level and RAID size of the destinationrange entry) is created and stored at an appropriate location such thatthe RAID information is appropriately distributed across data banks 110to ensure that the desired level of RAID is achieved with respect to theset of segments. It will be noted that in some cases the migrationprocess will entail the copying of segments between data banks 110, themovement of RAID data between data banks 110, the creation and storingof new RAID data from the set of segments being migrated, etc. and thatthe segments moved, RAID data created, stored or moved, will depend onboth the configuration of the source range and the desired configurationof the destination range.

Accordingly, during this migration process there will be an original setof data segments and corresponding redundancy segments configuredaccording to the source range entry and a new set of data segments andcorresponding redundancy segments configured according to thedestination range entry, where the migration process creates and storesthe data for the new set of data segments and corresponding redundancysegments. It will be noted that occasionally a data segment orredundancy segment may be associated with the same data bank 110 in bothconfigurations. In such cases it may not be necessary to create or movesuch a data or redundancy segment.

By keeping the original set of data segments and correspondingredundancy segments configured according to the source range entry whilemigrating these segments to the configuration described by thedestination range entry any errors or other failures that occur duringthe migration process may be dealt with. More specifically, if anyerrors or failures are detected during this migration process a “fallback” operation may take place, such that the original segments may beaccessed using the source range entry, and the original data segmentsand corresponding redundancy segments stored in accordance with thesource range entry, without loss or unavailability of data. Themigration process can then be started again using the source range entry(or some other range entry) as a starting point.

During this migration process, one or more commands may be received bydata banks 110. If these commands correspond to another volume orportions of the volume that are not currently being migrated thesecommands may be carried out substantially as described above. For thosereceived commands which correspond to the segments currently beingmigrated (in other words, the set of segments described by both thesource range entry and destination range entry and being moved, created,etc.) it can be determined what type of operation is being requested. Ifthe command is a READ command it may be carried out utilizing the sourcerange entry (in other words, the pertinent segment(s) may be located andread using the configuration described in the source range entry and theREAD command executed utilizing these original data and redundancysegments stored according to the source range entry). If, however, thecommand is a WRITE command the command may be carried out utilizing boththe source range entry and the destination range entry. In other words,the pertinent original segment(s) may be located and written accordingto the above procedure for a WRITE command using the configurationdescribed in the source range entry utilizing these original data andredundancy segments stored according to the source range entry,additionally, the pertinent original segment(s) may be located andwritten according to the above procedure for a WRITE command using theconfiguration described in the destination range entry utilizing thesenew data and redundancy segments stored according to the destinationrange entry.

After the completion of the migration of the set of segments from theoriginal data and redundancy segments stored according to the sourcerange entry to the new set of data segments and redundancy segmentsconfigured according to the destination range entry a “fall forward” mayoccur at step 1580. This “fall forward” operation may entail theintegration of the destination range entry into the table describing thevolume and the alteration of the Type designation of the destinationrange entry to indicate it is a normal range entry. This “fall forward”operation may be accomplished by copying values from the destinationrange entry to the source destination entry, the manipulation of memorypointers or some other operation entirely.

In any event, at the completion of the “fall forward” operation thetable for the volume will have a new range entry describing the samesegments as the source range entry (which has been replaced in the tabledescribing the volume by this new range entry), marked as a normal“Type” and describing the new configuration for the range (for example,new databank count, new databank order, new RAID level, new RAID size,etc.)

At some later point then, range entries (for example, one or more newrange entries as described above) having the same configuration may beconsolidated at step 1590. In one embodiment, if two or more rangeentries in a table corresponding to a volume have the same configuration(for example, same databank count, databank order, same RAID level, sameRAID size, etc.) but correspond to different sets of segments theserange entries may combined into one range entry. This combination may beaccomplished, for example, by altering the start and end entries of onerange entry to encompass the address of both the range entries anddeleting the other range entry. This consolidation process may occur asa background process the periodically evaluates the range entries in thevolume tables to determine if one or more range entries should beconsolidated.

To aid in an understanding of the above migration process, attention isnow directed to FIG. 16-21 which depict the tables and range entries inan example of one embodiment of a migration process. With respect tothis example, suppose initially the a volume has been created on adistributed RAID system comprising five databanks 110, where the volumecomprises 10,000 segments and the volume is divided into two ranges: afirst range of 5000 segments stored on four of the five databanks 110,with RAID 5 and a (3+1) parity set and a second range of 5000 segmentsstored on five of the five databanks 110, with RAID 5 and a (4+1) parityset.

Referring now to FIG. 16, one embodiment of a table 1650 for thisexample volume which may a part of tables 240 stored in each of datastores 250 of each data bank 110 in a distributed RAID system isdepicted. More specifically, the volume stored on the distributed RAIDsystem comprising five data banks 110, comprises 10,000 segment (asreflected by value of the Segment Count field of the table 1650) in tworanges (as reflected by the value of the Range Count field of the table1650). The first range is described by the first range entry 1624 awhich is a normal range entry (as reflected by the value of the Typefield of the first range entry 1624 a). The segments corresponding tothe first range (which comprise the segments of the volume correspondingto table 1650 with the addresses 0 to 4999 as reflected by the values ofthe Start and End fields of the first range entry 1624 a) are arrangedacross four of the five data banks according to the data bankpermutation “2, 4, 3, 1” (as reflected by the values of the DatabankCount and Databank Order fields in first range entry 1624 a) and RAID 5is implemented with respect to these segments with a parity size of 3+1(as reflected by the Network RAID level and Network RAID size fields infirst range entry 1624 a).

The second range is described by the second range entry 1624 b which isa normal range entry (as reflected by the value of the Type field of thefirst range entry 1624 b). The segments corresponding to the first range(which comprise the segments of the volume corresponding to table 1650with the addresses 5000 to 9999 as reflected by the values of the Startand End fields of the first range entry 1624 b) are arranged across allfive data banks 110 according to the data bank permutation “4, 2, 1, 3,5” (as reflected by the values of the Databank Count and Databank Orderfields in first range entry 1624 b) and RAID 5 is implemented withrespect to these segments with a parity size of 4+1 (as reflected by theNetwork RAID level and Network RAID size fields in first range entry1624 a). It will be noted with respect to FIG. 16 and the other FIGUREScorresponding to this example that only certain values of certainentries are discussed in order to illustrate this particular example,and the discussion of these entries and values should in no way be takenas restriction on the presence or values of other entries which may ornot be in such tables.

Now suppose that a data bank 110 is added to the distributed RAID systemon which the volume is stored, bringing the number of data banks 110 inthe distributed RAID system to six. Further suppose that as a result ofthe addition of the sixth data bank 110, the volume represented by table1650 is selected for migration and that moreover, the second rangerepresented by range entry 1624 b has been selected for migration withinthis volume (STEPS 1505, 1510). At this point it will be determined thatthere are a different number of data banks 110 (STEP 1520). Assuming nowthat there is space in each of the data stores 250 of the six data banks110 of the distributed RAID system to store the segments andcorresponding redundancy data associated with the second range, the newDatabank Count determined for the destination range (after migration)will be six and a new Databank Order for the migration of the secondrange may be determined (STEP 1530). For purposes of this example,suppose that the newly determined Databank Order is “1, 6, 2, 3, 4, 5”.Next, the RAID level and RAID size for the destination range may bedetermined. Again, assume that no change has been made to the level ofRAID desired, thus the RAID level will still be 5, however, as one moredata bank 110 has been added, the RAID size (in this case, parity size)for this RAID level may now be changed to (5+1).

Once the databank order, databank count, RAID level and RAID size havebeen determined (one or more of which may be the same as the existingrange of the volume), the source range of the source volume ispartitioned (STEP 1550). Turning to FIG. 17, one embodiment of the table1650 for this example volume after the partitioning of the second rangeis depicted. Note before discussing in more detail that as the firstrange entry 1624 a is not being discussed further, the entriescorresponding to the first range entry 1624 a have been condensed andthus do not appear in FIG. 17 or in the remainder of the FIGUREScorresponding to this example.

In the present example, the range being migrated originally comprised5000 segments having the addresses 5000-9999. Assuming for purposes ofthis example a migration size of 1000, new range entry 1624 c will becreated in table 1650 where the new range entry 1624 c corresponds to1000 segments of the original range (as reflected by the Start (5000)and End (5999) fields of the range entry 1624 c) New range entry 1624 calso reflects the configuration of the original range (in other words)before migration (as reflected by the Databank Count (5), Databank Order(“4, 2, 1, 3, 5”), Network RAID level (5) and Network RAID size (4+1)fields of the range entry).

Similarly the range entry 1624 b corresponding to the original range andthe table 1650 describing the volume may be updated to reflect thepartition. Specifically, the Start (6000) and End fields (9999) of theoriginal range entry 1624 b may be changed to reflect the partition ofthe 1000 segments of the original range entry into newly created rangeentry 1624 c while the Range Count (3) filed of the table may also bealtered to reflect this partition (in this case the partition increasedthe number of range entries by one). The newly created range entry 1624c may then be marked as a source for purposes of the data migration (asreflected by the Type field (Source) of the new range entry 1624 c.

Notice then, that with respect to the two range entries 1624 b and 1624c both the original range entry 1624 b and the newly created range entry1624 c, while having been configured to reflect the partitioning of the1000 segments into the new range entry 1624 c, in other respects stillreflects the configuration of the original range entry 1624 b. Thedifference being that the configuration of the original range entry 1624b indicates that the segments corresponding to the original range entry1624 b are not undergoing migration (for example, the Type (Normal)field of original range entry 1624 b). In contrast, new range entrycorresponding to the 1000 segments previously corresponding to theoriginal range entry 1624 b reflects that these 1000 segments arecurrently undergoing migration (for example, the Type (Source) filed ofthe new range entry 1624 c).

Once the original range has been partitioned to create a source rangeentry (here new range entry 1624 c) a destination range entrycorresponding to the source range entry may be created and configured.Turning to FIG. 18, one embodiment of the creation of a destinationrange entry for this example is depicted. Here destination range entry1626 is a duplicate of the new range entry 1624 c (the value of thefields is the same) with the exception that the Type field of thedestination range entry 1626 is set to “Destination”. Thus, there nowexists new range entry 1624 c (referred to also as the source rangeentry) which is a part of table 1650 describing the range beingmigrated, and the destination range entry 1626 which may, in oneembodiment, be separate from table 1650 or associated with table 1650 ornew range entry 1624 c.

The destination range entry may then be configured by assigning thedestination range entry 1626 the values corresponding to the databankcount, databank order, RAID level and RAID size which were determinedabove for the corresponding range of the migrated volume. At this pointthen, the source range entry describes the currently implementedconfiguration for example, the databank count, databank order, RAIDlevel, RAID size, etc.) associated with a set of segments (equal innumber to the migration segment size) of the original range, while thedestination range entry describes the configuration associated with thesame set of segments as it is to be implemented in the migrated volume.

Thus, there now exists new range entry 1624 c (the source range entry)which is a part of table 1650 describing the old configuration of therange being migrated, and the destination range entry 1626 which may, inone embodiment, be separate from table 1650, and which described therange being migrated in the new configuration (in other words, theconfiguration of the segments described by the range entry 1626 (andassociated redundancy segments) after the migration process).

Using the source range entry 1624 c and the destination range entry1626, the set of segments may then be migrated from the configurationdescribed by the source range entry to the configuration described bythe destination range entry 1626 (STEP 1570). This migration process mayensure that the set of segments are stored according to the databankorder of the destination range entry 1626 and that any datacorresponding to the implementation of RAID in conjunction with thedestination range 1626 (in other words according to the RAID level andRAID size of the destination range entry 1626) is created and stored atan appropriate location such that the RAID information is appropriatelydistributed across data banks 110 to ensure that the desired level ofRAID is achieved with respect to the set of segments.

Accordingly, during this migration process there will be an original setof data segments and corresponding redundancy segments configuredaccording to the source range entry 1624 c stored on data banks 110 anda new set of data segments and corresponding redundancy segmentsconfigured according to the destination range entry 1626 c stored ondata banks 110, where the migration process creates and stores the datafor the new set of data segments and corresponding redundancy segments.By keeping the original set of data segments and correspondingredundancy segments configured according to the source range entry 1624c while migrating these segments to the configuration described by thedestination range entry 1626 any errors or other failures that occurduring the migration process may be dealt with. More specifically, ifany errors or failures are detected during this migration process a“fall back” operation may take place, such that the original segmentsmay be accessed using the source range entry and the original datasegments and corresponding redundancy segments stored in accordance withthe source range entry without loss or unavailability of data. Themigration process can then be started again using the source range entry(or some other range entry) as a starting point.

Assuming no error occur during this migration process, however, a “fallforward” process may occur after the completion of the migration of theset of segments from the original data and redundancy segments storedaccording to the source range entry 1624 c to the new set of datasegments and redundancy segments configured according to the destinationrange entry 1626 (STEP 1580). In other words, at this point there existthe original set of segments and corresponding redundancy segmentsconfigured according to source range entry 1624 c and the new set ofsegments and corresponding redundancy segments created during themigration process to conform to configuration described by destinationrange entry 1626. This “fall forward” operation may be initiated suchthat the new set of segments and corresponding set of redundancysegments may be utilized with respect to commands received by the databanks 110. In other words, that command may now be carried exclusivelywith respect to the new segments and corresponding redundancy segments.

Thus, this “fall forward” operation may entail the integration of thedestination range entry 1626 into the table 1650 describing the volumeand the alteration of the Type designation of the destination rangeentry 1626 c to indicate it is now a normal range entry (in other wordsthe corresponding segments are not undergoing migration and may benormally accessed). In one embodiment, the “fall forward” operation maybe accomplished by copying values from the destination range entry 1626to the source destination entry 1624 c, the manipulation of memorypointers or some other operation entirely. In any event, at thecompletion of the “fall forward” operation the table 1650 for the volumewill comprise a new range entry describing the same segments as thesource range entry 1624 c (which has been replaced in the table 1650describing the volume by the new range entry), marked as a normal “Type”and describing the destination configuration (for example, new databankcount, new databank order, new RAID level, new RAID size, etc.)

Turning to FIG. 19, one embodiment of a table 1650 after a fall forwardoperation in the preceding example is depicted. Here, table 1650 nowcomprises new range entry 1624 d which has replaced third range entry1624 c (depicted in FIG. 18). Note that new range entry 1624 d describesthe same set of segments in the same configuration as described by thedestination range entry 1626 (for example, new range entry 1624 ddescribes the same segment addresses 5000-5999, the same network RAIDlevel (5), the same network RAID size (5+1), the same Databank Count(6), the same Databank Order (“1, 6, 2, 3, 4, 5”, etc.) but, in contrastto destination range entry 1626, the Type of new range entry 1624 d hasthe value “Normal”. Accordingly, as the set of segments andcorresponding redundancy segments are stored on data banks 110 in theconfiguration described in new range entry 1624 d, the configuration ofthe set of segments and corresponding redundancy segments is describedby new range entry 1624 d into table 1650 describing the volume, thesesegments and corresponding redundancy segments may be accessed normallyusing new range entry 1624 d. Thus, the migration process for these setof segments is completed. The migration process may then be carried outwith respect to the next set of segments (for example, of the nextmigration segment size of segments).

It may be realized after reading the above that once the migrationprocess has been carried out multiple times there may be a number ofrange entries in a table describing a volume where each of these rangeentries correspond to different segments (in other words sets ofsegments with different addresses), but each of the sets of segments isconfigured similarly (for example, according to the new configuration,including the same random permutation of data banks 110, same number ofdata banks 110, same RAID level, etc.). In this case, it may bedesirable after a certain time period to consolidate one or more rangeentries in a table describing a volume, where consolidating these rangeentries comprises combining these range entries into a single rangeentry which describes both set of segments and the same configuration.

The consolidation of range entries may be better understood withreference to FIGS. 20-21 which depict one example of this consolidationprocess in accordance with a continuation of the example of FIGS. 16-19.More specifically, FIGS. 20A and 20B depicts table 1650 after anothermigration has taken place with respect to the set of segmentscorresponding to addresses 6000-6999. Thus, at this point, new rangeentry 1624 e has just been created in table 1650 where new range entry1624 e corresponds to the set of segments 6000-6999 and describes theconfiguration of this set of segments and the corresponding redundancysegments (for example, new range entry 1624 e describes the segmentaddresses 6000-6999, network RAID level (5), network RAID size (5+1),Databank Count (6), Databank Order (“1, 6, 2, 3, 4, 5”, etc.).

Additionally, as described above the previously added range entry 1624 ddescribes a previously migrated set of segments in the sameconfiguration as described by the new range entry 1624 e(for example,range entry 1624 d describes a different segment addresses 5000-5999,the same network RAID level (5), the same network RAID size (5+1), thesame Databank Count (6), the same Databank Order (“1, 6, 2, 3, 4, 5”,etc.). Accordingly, as range entry 1624 e and 1624 e correspond todifferent sets of segments and corresponding redundancy segmentsconfigured according to the same configuration it may be desired toconsolidate range entry 1624 d and range entry 1624 e.

FIG. 21 depicts the resulting table 1650 after this configurationprocess. Notice with respect to table 1650 that range entry 1624 d andrange entry 1624 e have been combined into range entry 1624 f. As can beseen then, range entry 1624 f now corresponds to both sets of segmentspreviously corresponding to range entry 1624 d and range entry 1624 e(as reflected by the Start value (5000) and End value (6999) of rangeentry 1624 f) and describes the same configuration as both range entry1624 d and range entry 1624 e (for example, network RAID level (5),network RAID size (5+1), Databank Count (6), Databank Order (“1, 6, 2,3, 4, 5”), etc.). Thus, both sets of segments and correspondingredundancy segments previously described and accessed using rangeentries 1624 d and 1624 e may now be accessed using a single range entry1624 f. It will be apparent that the consolidation of range entries mayoccur either synchronously or asynchronously to other operations of adistributed RAID system, a data bank 110, a distributed RAID application210, including synchronously or asynchronously to the migration of otherranges, volumes, etc. Additionally, it will be apparent that such aconsolidation process may occur at almost any time interval desired orbe triggered by almost any event desired, including for example, thecompletion of the migration of a certain number of segments, etc.

After a review of the above description of the migration process, itwill be understood how such a migration process may be utilized inconjunction with a variety of potentialities which may occur during theoperation of a distributed RAID system, including the addition orremoval of a data bank 110 from the distributed RAID system, a desiredchange in RAID level for one or more of the volumes stored in thedistributed RAID system, the addition or removal of a storage from adata store 250 of a data bank 110, etc.

There is, additionally, another situation which may present duringoperation of the distributed RAID system where the migration process asdescribed above may be also be useful. This situation may be related tothe failure of a disk drive within a data store 250 of a data bank 110.Thus, before discussing the migration process involved with failure of adisk drive it may be helpful to first discuss one embodiment of sparingprovisions which may be made with respect to a distributed RAID system.

To that end, attention is directed to FIG. 22 which depicts oneembodiment of a distributed RAID system which includes a set of databanks 110, each having a data store 250 comprising a set of disks 250.In many cases a mean time between failure for disks 252 may berelatively high with respect to other components of the distributed RAIDsystem. Thus, an efficient methodology for recovering from the failureof a disk 252 in a distributed RAID system may be desired.

Accordingly, in one embodiment, a sparing portion 2202 of one or moredisks 252 on one or more of the data stores 250 of each data bank 110may be reserved (for example, remain unallocated or unutilized in normaloperation), where the aggregate size of the sparing portions 2202 may beequal to the largest disk 252 in all of the data stores 250. Moreparticularly, in one embodiment, each disk 252 of each data store 250may have an equally sized sparing portion 2202 reserved, where theaggregate size of all the sparing portions 2202 of each disk 252 in eachdata reserve may equal to, or larger than, the size of a disk 252.Expressed mathematically, if the number of disks 252 comprising all ofthe data stores 250 is N and the size of each of the disks 252 is X thesparing portion 2202 of each disk 250 may be approximately equal to X/Nor X/(N−1). For a numerical example, suppose the each of the disks 252in the data stores 250 are all 1 TB in size. In this case, the aggregatesize of the sparing portions 2202 of each disk 252 (or one less than thenumber of disks 252) may be equal to approximately 1 TB.

By reserving this sparing portion 2202 of each disk 252, whenever a disk252 fails this failure can be accounted for by distributing the datastored on the failed disk 252 across each of the sparing portions 2202reserved on each of the non-failed disks 252 in each of data store 250,if needed. More specifically, when a failure of a disk 252 in a datastore 250 occurs, a volume stored on that disk 252 may be migrated suchthat the data segments and corresponding redundancy segments of thatvolume stored in other disks 252 of the distributed RAID system may notneed to be migrated, but that the segments stored on the failed disk 252will be migrated to the sparing portions 2202 of the other disks 252 inthe distributed RAID system.

In one embodiment, the migration of a volume in conjunction with a diskfailure may be accomplished in this manner through a selection of anappropriate permutation of data banks 110 (in other words, the order ofdata banks 110 for the new configuration of the volume may be selectedto minimize transfer of segments which were not stored on the faileddisk 252). Additionally, by tracking the number of disks 252 in eachdata bank 110 used to store the new volume the segments associated withthe volume which were stored on the failed disk 252 may be effectivelymigrated to the sparing portion 2202 of each disk 252.

To illustrate with an actual example, assume for a moment that adistributed RAID system comprises three data banks 110, each data bank110 having a data store 250 comprising a set of twelve disks 252, eachdisk 252 1 TB in size. Assume further that a volume is stored on thisexample of a distributed RAID system, where the volume is represented bythe example table 2350 depicted in FIG. 23. Note with respect to table2350 that initially, the volume represented by table 2350 is stored onall three data banks 110 of the distributed RAID system. Specifically,the volume has one range (represented by range entry 2324) which isstored according to the Databank Order “2, 3, 1” where the data bank “2”has a Disk Count of 12 and a Disk Order of “2, 4, 3, 1, 5, 6, 8, 7, 9,11, 10 12” meaning that segments of associated with the volume which areto be stored on data bank “2” are stored on the disks 252 in the datastore 250 of data bank “2” according to this order.

In other words, the first segment to be stored on data bank “2” isstored on disk “2” of data store 250 of data bank “2”, the secondsegment to be stored on data bank “2” is stored on disk “4” of datastore 250 of data bank “2,” etc. The thirteenth segment to be stored ondata bank “2” is stored on disk “2” of data store 250 of data bank “2,”the second segment to be stored on data bank “2” is stored on disk “4”of data store 250 of data bank “2”, etc. Segments associated with thevolume are similarly stored on the twelve disks 252 of data bank “3”according to the corresponding Disk Order “11, 7, 6, 2, 3, 4, 9, 8, 10,5, 1, 12” and on the twelve disks 252 data bank “1” according to thecorresponding Disk Order “5, 7, 6, 11, 12, 3, 4, 9, 8, 10, 5, 2, 1.”

Suppose now, that after operating for a period of time in conjunctionwith the volume represented by the table 2350, a disk 252 on one of thedata banks 110 of the distributed RAID system fails. Specifically,suppose that disk “9” of data bank “3” fails. In this case, the dataassociated with the volume represented by table 2350 may be migrated tothe sparing portions 2202 of each of the 12 disks 252 of data banks “2”and “1” and 11 disks of data bank “3”.

Specifically, in one embodiment, this may be done by selecting valuesfor the configuration of the new volume which may entail that during themigration process for the volume only segments stored on disk “9” ifdata bank “3” may need to be migrated to other locations. It will benoted that as the segments to be migrated to other locations in thisexample reside on a failed disk, the migration of these segments mayentail the recreation of the segments (for example, from a redundancysegment or, in the case of redundancy segments, from the data segmentson which the redundancy segment is based) before the segment is storedin its new location (for example, a sparing portion 2202 of a disk 252).

Thus, after the migration process for the example volume described abovethe table representing the volume may be similar to table 2450 depictedin FIG. 24. Specifically, the volume still has one range (represented byrange entry 2424) which is stored according to the Databank Order “2, 3,1” where the data bank “2” has a Disk Count of 12 and a Disk Order of“2, 4, 3, 1, 5, 6, 8, 7, 9, 11, 10 12” meaning that segments ofassociated with the volume which are to be stored on data bank “2” arestored on the disks 252 in the data store 250 of data bank “2” accordingto this order. Segments associated with the volume are similarly storedon the twelve disks 252 data bank “1” according to the correspondingDisk Order “5, 7, 6, 11, 12, 3, 4, 9, 8, 10, 5, 2, 1.” However, noticenow that the Disk Count associated with data bank “3” now has the valueof “11” as one of the disks 252 of data bank “3” has failed and the DiskOrder “11, 7, 6, 2, 3, 4, 9, 8, 10, 5, 1, 12” associated with data bank“3” no longer includes disk “9” as disk “9” has failed. Thus, thesegments of the migrated volume may b stored according to theconfiguration depicted in table 2424 after the migration process causedby the failure of disk “9” on data bank “3” of the distributed RAIDsystem.

With respect to the discussion above regarding migration of data inresponse to a failure of a disk 252, in the example discussed data wasmoved onto sparing portions 2202 of operational disks 252 in thedistributed RAID system, however, in cases where there is free storageon disks 252 of the distributed RAID system which is not sparing portion2202, the segments stored on the failed disk may be migrated to thisfree storage on each of the disks 252. Thus, there may be cases wheresegments of a volume on a failed disk 252 may be migrated to areas ofother disks 252 on data stores 250 of data banks 110 which are not thesparing portion 2202 or cases where some segments on a failed disk maybe migrated to areas of other disks 252 on data stores 250 of data banks110 which are not the sparing portion 2202 while other segments whichresided on the failed disk may be migrated to the sparing portions 2202of other disks 252. Regardless of the location on a disk 252 to whichthe segments on the failed disk are migrated the migration process mayoccur substantially similarly to the embodiments of the migrationprocess discussed above.

In any event, once a failed disk 252 has been replaced it may be desiredto rebuild the volume. This rebuilding may entail a migration process bywhich the segments of the volume (originally stored on the failed disk252) which were migrated to locations on the other disks 252 of thedistributed RAID system are migrated back to the disk 252 which wasreplaced. In one embodiment, the rebuild of a volume in conjunction witha disk replacement may be accomplished substantially in reverse to thisprocess described above through a selection of an appropriate DatabankOrder, Disk Counts and Disk Orders. Thus, after the replacement of afailed disk 252 and the rebuilding process, a volume may substantiallyresemble its original configuration before the disk failure.

Another important point to note with respect to the above description isthat in many cases segments associated with multiple different volumesmay be stored on a failed disk. Thus, the order in which the segmentscorresponding to each of those volumes is migrated to other disks 252 ofthe distributed RAID system in the case of a disk failure or the orderin which the segments corresponding to the volume are rebuilt (forexample, restored to their original configuration) when a disk isreplaced may be of some importance, as various volumes may containrelatively more important data, may be accessed with greater frequency,etc.

Accordingly, in one embodiment the different performance characteristicsassociated with different volumes may be taken into account inconjunction with a migration process corresponding to a volume. Inparticular, in one embodiment, these different performancecharacteristics may be accounted for during a rebuilding process formultiple volumes whose segments previously resided on a failed disk andwhere each of these volumes is being migrated in conjunction with arebuilding process after this failed disk is replaced. While embodimentsof accounting for these performance characteristics will be described inconjunction with such rebuilding processes it should be noted that otherembodiments may be equally well applied in other contexts which entailmigration.

FIG. 25 depicts one embodiment of a method for selecting volumes tomigrate. At step 2510 a migration process comprising multiple volumesmay be initiated. This process may be initiated in response to a failureof a disk 252, such that segments associated with multiple volumes mayneed to be migrated to other disks in the distributed RAID system; theprocess may be initiated in response to the replacement of a failed disksuch that segments associated with multiple volumes may need to bemigrated from their current storage location back to at least thereplacement disk 252; or the process may be initiated in response to avariety of other occurrences such as the addition or removal of databanks 110 from the distributed RAID system, a change of RAID level formultiple volumes, the installation of or removal of disks 252 of a datastore, some combination of these events, or some other occurrenceentirely.

Based on the migration process initiated, at step 2520 it can bedetermined which volumes will be involved in the initiated migrationprocess. This determination may be made, for example, based on whichvolumes have segments which are stored on a failed disk or a removeddata bank, which volumes have segment that are to be moved to an addeddata bank or disk, or to a replacement disk, etc.

After the volumes impacted by the initiated migration process have beendetermined, the volumes can be prioritized at step 2530 based upon avariety of criteria. These criteria may include for example, such thingsas a quality of service designation associated with a volume or a numberof accesses to the volume or some combination of these criteria.

More specifically, in one embodiment a quality of service indicator maybe stored in a table corresponding to the volume (for example, qualityof service indicator 514 of table 550 of FIG. 5). This quality ofservice indicator may indicate a relative level priority with respect toother volumes having a different quality of service indicator maintainedby the distributed RAID system. For example, a volume whose table hasquality of service indicator of “1” may be relatively higher prioritywith respect to a volume whose table has a quality service indicator of“2”. This quality of service indicator may, for example, be set when auser defines the volume, as discussed above. This quality of serviceindicators associated with each of the impacted volumes may be used toprioritize these volumes

Another criterion which may be used to prioritize the impacted volumesmay be the number of access to the volumes. In one embodiment, a tableof tables 240 on each data bank 110 may track the number of accesses toeach volume over a period of time (minute, hour, day, etc.). The numberof accesses to each volume may be used to prioritize the volumes. Forexample, the volume with the highest number of accesses may be assignedthe highest priority and the volume with least number of accesses may beassigned the lowest priority.

It will be apparent that almost any criterion desired may be used toprioritize the volumes and that combinations of these criteria may beutilized as well. For example, the impacted volumes may be initiallyprioritized based upon a quality of service indicator associated withthe volume and amongst volumes which have the same quality of serviceindicator associated with them, they may be prioritized based uponnumber of accesses to the volume over a given time period.

Once the impacted volumes have been prioritized, the migration processfor these volumes may be carried out at step 2540. Specifically, themigration process may be done in conjunction with the priority of thatvolume, such that higher priority volumes may be migrated before thosevolumes with lower priority levels. By migrating the volumes accordingto the priority levels, higher priority volumes may be rebuilt orotherwise achieve a higher level of functionality before those volumeswith lower priority.

In the foregoing specification, the invention has been described withreference to specific embodiments. However, one of ordinary skill in theart appreciates that various modifications and changes can be madewithout departing from the scope of the invention as set forth in theclaims below. Accordingly, the specification and figures are to beregarded in an illustrative rather than a restrictive sense, and allsuch modifications are intended to be included within the scope ofinvention.

Benefits, other advantages, and solutions to problems have beendescribed above with regard to specific embodiments. However, thebenefits, advantages, solutions to problems, and any component(s) thatmay cause any benefit, advantage, or solution to occur or become morepronounced are not to be construed as a critical, required, or essentialfeature or component of any or all the claims.

1. A system, comprising: a plurality of storage appliances, each storageappliance coupled to each of the other storage appliance, wherein eachstorage appliance includes: a processor; a data store; a computerreadable medium comprising instructions executable to: select a volumefrom a set of volumes wherein the volume and redundancy datacorresponding to the volume are stored on each of the plurality ofstorage appliances such that a set of data segments corresponding to thevolume and a corresponding set of redundancy segments are stored on theplurality of storage appliances according to a first configurationcomprising a first RAID level, a first RAID size, a first storageappliance count and a first storage appliance Order; and migrate thevolume from the first configuration, wherein migration of the volumecomprises storing at least a portion of the volume on each of theplurality of storage appliances according to a second configurationcomprising one or more of a second RAID level different from the firstRAID level, a second RAID size different from the first RAID size, asecond storage appliance count different from the first storageappliance count or a second storage appliance order different from thefirst storage appliance order.
 2. The system of claim 1, whereinmigrating the volume comprises migrating both the set of data segmentsof the portion of the volume and the set of redundancy segmentscorresponding to the set of data segments of the portion of the volume.3. The system of claim 1, wherein the second RAID level, second RAIDsize, second storage appliance count or second storage appliance orderare selected such that only the set of data segments are moved whenmigrating the volume.
 4. The system of claim 1, wherein the migration isinitiated in response to an addition of a storage appliance to theplurality of storage appliances, a removal of a storage appliance fromthe plurality of databanks, a failure of a storage appliance of theplurality of storage appliances, an addition of storage to the datastore of at least one of the plurality of storage appliances, a removalof storage from the data store of at least one of the plurality ofstorage appliances, a failure of storage in the data store of at leastone of the plurality of storage appliances or a change in a level ofRAID for at least a portion of the first volume.
 5. The system of claim1, wherein a portion of the data store on each storage appliance isreserved to perform migration.
 6. The system of claim 1, wherein asource range entry and a destination range entry are used during themigration of the portion of the volume.
 7. The system of claim 6,wherein the destination range entry replaces the source range entry atthe completion of the migration.
 8. A method for distributed RedundantArray of Independent Disks (RAID) implemented on a plurality of storageappliances, comprising: storing a volume and corresponding redundancydata on each of the plurality of storage appliances, such that a set ofdata segments corresponding to the volume and a corresponding set ofredundancy segments are stored on the plurality of storage appliancesaccording to a first configuration comprising a first RAID level, afirst RAID size, a first storage appliance count and a first storageappliance Order; selecting the volume for migration; and migrating thevolume from the first configuration, wherein migration of the volumecomprises storing at least a portion of the volume on each of theplurality of storage appliances according to a second configurationcomprising one or more of a second RAID level different from the firstRAID level, a second RAID size different from the first RAID size, asecond storage appliance count different from the first storageappliance count or a second storage appliance order different from thefirst storage appliance order.
 9. The method of claim 8, whereinmigrating the volume comprises migrating both the set of data segmentsof the portion of the volume and the set of redundancy segmentscorresponding to the set of data segments of the portion of the volume.10. The method of claim 8, wherein the second RAID level, second RAIDsize, second storage appliance count or second storage appliance orderare selected such that only the set of data segments are moved whenmigrating the volume.
 11. The method of claim 8, wherein the migrationis initiated in response to an addition of a storage appliance to theplurality of storage appliances, a removal of a storage appliance fromthe plurality of databanks, a failure of a storage appliance of theplurality of storage appliances, an addition of storage to the datastore of at least one of the plurality of storage appliances, a removalof storage from the data store of at least one of the plurality ofstorage appliances, a failure of storage in the data store of at leastone of the plurality of storage appliances or a change in a level ofRAID for at least a portion of the first volume.
 12. The method of claim8, wherein a portion of the data store on each storage appliance isreserved to perform migration.
 13. The method of claim 8, wherein asource range entry and a destination range entry are used during themigration of the portion of the volume.
 14. The method of claim 13,wherein the destination range entry replaces the source range entry atthe completion of the migration.
 15. A computer readable mediumcomprising computer executable instructions for implementing distributedRedundant Array of Independent Disks (RAID) on a plurality of storageappliances, the computer executable instructions executable for:selecting a volume from a set of volumes wherein the volume andredundancy data corresponding to the volume are stored on each of theplurality of storage appliances such that a set of data segmentscorresponding to the volume and a corresponding set of redundancysegments are stored on the plurality of storage appliances according toa first configuration comprising a first RAID level, a first RAID size,a first storage appliance count and a first storage appliance Order; andmigrating the volume from the first configuration, wherein migration ofthe volume comprises storing at least a portion of the volume on each ofthe plurality of storage appliances according to a second configurationcomprising one or more of a second RAID level different from the firstRAID level, a second RAID size different from the first RAID size, asecond storage appliance count different from the first storageappliance count or a second storage appliance order different from thefirst storage appliance order.
 16. The computer readable medium of claim15, wherein migrating the volume comprises migrating both the set ofdata segments of the portion of the volume and the set of redundancysegments corresponding to the set of data segments of the portion of thevolume.
 17. The computer readable medium of claim 15, wherein the secondRAID level, second RAID size, second storage appliance count or secondstorage appliance order are selected such that only the set of datasegments are moved when migrating the volume.
 18. The computer readablemedium of claim 15, wherein the migration is initiated in response to anaddition of a storage appliance to the plurality of storage appliances,a removal of a storage appliance from the plurality of databanks, afailure of a storage appliance of the plurality of storage appliances,an addition of storage to the data store of at least one of theplurality of storage appliances, a removal of storage from the datastore of at least one of the plurality of storage appliances, a failureof storage in the data store of at least one of the plurality of storageappliances or a change in a level of RAID for at least a portion of thefirst volume.
 19. The computer readable medium of claim 15, wherein aportion of the data store on each storage appliance is reserved toperform migration.
 20. The computer readable medium of claim 15, whereina source range entry and a destination range entry are used during themigration of the portion of the volume.
 21. The computer readable mediumof claim 20, wherein the destination range entry replaces the sourcerange entry at the completion of the migration.